Member of Technical Staff Pre-training Systems

You will design and operate distributed infrastructure for large-scale model training. You will scale training across GPU clusters, optimize communication and synchronization, improve checkpointing and recovery, eliminate system bottlenecks, and improve reproducibility and hardware utilization.

Responsibilities

• Scale distributed training across large GPU clusters using data, tensor, and pipeline parallelism
• Optimize communication patterns and gradient synchronization
• Improve checkpointing, fault tolerance, and job recovery systems
• Profile and eliminate performance bottlenecks across compute, networking, and storage
• Improve experiment reproducibility and orchestration workflows
• Increase hardware utilization and training throughput
• Collaborate with Kernels and Research to align model architecture with systems realities

Requirements

• Strong software engineering and distributed systems fundamentals
• Experience training large models in multi-node GPU environments
• Deep understanding of parallelism strategies and performance trade-offs
• Experience debugging cross-layer issues in production ML systems
• Ability to operate critical infrastructure
• Track record of improving performance or reliability of large-scale systems

Benefits

• Equity as a significant part of total compensation
• 401(k) plan with 6% salary matching
• Health, dental, and vision insurance for employees and dependents
• Unlimited paid time off
• Visa sponsorship
• Relocation stipend to San Francisco