Performance Engineer

ABOUT CUSPAI CuspAI is the frontier AI company on a mission to solve the breakthrough materials needed to power human progress. While nature took billions of years to perfect molecules, we are harnessing AI to unlock trillion-dollar materials breakthroughs in months, not millennia. Our founding team is the most cited in the world, comprised of world-class researchers in AI, chemistry and engineering. We are working on some of the hardest and most important challenges including energy, clean water, the future of compute, and carbon capture, and this is just the start of what our 'search engine' for next-generation materials will unlock. We invite you to be part of a diverse, innovative team at the intersection of AI and materials science, working to create impactful partnerships that drive innovation, scalability, and industry collaboration. This work matters. Your work matters. We’re on the cusp of the on-demand materials era. Join us. THE ROLE Due to the rapid scaling of our compute requirements, we are seeking a Performance Engineer to serve as the vital bridge between our frontier AI models and our high-performance computing infrastructure. YOUR IMPACT In this role you will be building the optimization layers and profiling infrastructure that maximizes the efficiency of our large-scale training and inference workloads, which is critical for accelerating our materials discovery pipeline and ensuring our researchers can iterate at the speed of thought.   At CuspAI, compute is our most precious resource. Your main focus initially will be profiling our current training runs to identify bottlenecks in Model FLOP Utilization (MFU). Over time you are likely to get involved in designing custom kernels and low-level optimizations that allow us to train larger, more complex models for chemistry and physics that were previously computationally prohibitive. WHAT YOU WILL DO Training & Inference Optimization - Analyze model architectures to optimize how mathematical operations map onto hardware, specifically focusing on NVIDIA GPU kernels and collective communication layers (NCCL). - Implement state-of-the-art efficiency techniques such as quantization, model pruning, and distillation to lower latency for our materials “search engine” APIs. - Streamline data pipelines to ensure high-performance accelerators are never “starved” for data during massive training runs. Profiling, Observability & Cost Management - Use advanced observability tools to trace end-to-end execution of training jobs, pinpointing exact locations of processing clogs or interconnect contention. - Build and maintain real-time dashboards tracking vital system metrics, including MFU, memory bandwidth utilization, and cluster uptime. - Monitor and justify the utilization of our AI accelerators (e.g. H100 clusters), preventing idle time and memory leaks to ensure every dollar of compute spend translates into scientific progress. - Develop scaling laws and internal tools to predict system behavior across compute, memory, and networking as our models grow in parameter count. Interdisciplinary Collaboration - Work closely with AI Researchers to adapt model architectures for better hardware compatibility without sacrificing scientific accuracy. - Partner with the Infrastructure team to influence the design of our future compute clusters based on performance data and profiling results. MUST HAVE SKILLS AND QUALIFICATIONS: - You are someone who gets excited about the opportunity to enable scientists to work on world changing challenges in this domain, with a personal interest in the potential applications of the technology that Cusp is building. - You have 3+ years of experience in performance engineering, site reliability engineering, or low-level software development, specifically focused on high-performance computing (HPC) or AI workloads. - Expert-level knowledge of GPU architecture and experience optimizing CUDA kernels or using libraries like Triton. - Deep understanding of distributed training frameworks (e.g. PyTorch Distributed, DeepSpeed, or Megatron-LM) and networking protocols. - Proficiency in profiling tools such as Nsight Systems, Nsight Compute, or Weights & Biases for performance tracking. - A "builder" mindset with the ability to work autonomously in an early-stage, fast-paced startup environment. BONUS POINTS (BUT NOT CRITICAL): - Experience working with specialized AI hardware such as TPUs or alternative accelerators. - An academic background or personal interest in Materials Science, Chemistry, or Physics. - Knowledge of MLOps practices, specifically regarding the deployment of models at scale in cloud environments (AWS/GCP/Azure). - Contributions to open-source performance optimization or machine learning libraries. ADDITIONAL CONSIDERATIONS This role could be based in our Cambridge, London, Amsterdam or Berlin offices, with the expectation of being in the office three days per week. Additionally, there may be regular travel required to other locations for collaboration and project work. As part of your onboarding, you will be expected to spend at least a few weeks in our Cambridge/London office learning about our platform and getting to know us in person. Additionally, there may be regular travel required to Europe and other regional and global locations for collaboration and project work. WHAT WE OFFER - A competitive salary: We value and reward impact and growth - Equity in CuspAI: You have a stake in the success of the company - Time off to stay fresh: 28 days holiday (DE, NL, UK) or 21 days holiday (JP, SG, US), in addition to local public holidays - ‘Gold Standard’ parental leave: 26 weeks (primary caregiver) and 12 weeks (secondary caregiver) at full pay - we look after you and your family while we work on the most important materials discovery problems together - Professional development budget: We invest in your career development so you can stay up to date with the latest industry knowledge or add to your skills to increase impact and growth - Solve meaningful problems: See how your work has a direct impact on advancing materials science and solving sustainability and climate-related problems through the creation and application of bleeding-edge SOTA technology and revolutionary techniques - True interdisciplinary teamwork: Be part of a deeply collaborative environment bridging AI research, computational chemistry, and experimental science - work with world-class researchers and engineers who enjoy sharing knowledge and supporting each other Join us in shaping the future of materials with AI. Together, we can create groundbreaking solutions for a more sustainable world. CuspAI is an equal opportunities employer committed to building a diverse and inclusive workplace. We do not discriminate on the basis of sex, race, religion or belief, ethnic or national origin, disability, age, citizenship, marital, domestic or civil partnership status, sexual orientation, gender identity, pregnancy or related condition (including breastfeeding), veteran status, or any other basis protected by applicable law. We actively encourage applications from all backgrounds and value the unique perspectives and contributions that diversity brings to our team. Please let us know if you require any specific adjustments during or after the interview process. We will do everything we can within reason to accommodate.