Member of Technical Staff Distributed Training Engineer

You will design and optimize distributed systems for large-scale training. You will implement parallelism and sharding strategies, improve distributed efficiency, remove data-loading bottlenecks, develop checkpointing, and create monitoring, profiling, and debugging tools.

Responsibilities

• Design and build reliable systems for large training runs
• Build scalable distributed training infrastructure for GPU clusters
• Implement and tune parallelism and sharding strategies
• Optimize distributed efficiency
• Build data-loading systems for multimodal datasets
• Develop checkpointing mechanisms
• Create monitoring, profiling, and debugging tools

Requirements

• Experience building distributed training infrastructure with PyTorch Distributed DDP or FSDP, DeepSpeed ZeRO, or Megatron-LM TP or PP
• Experience diagnosing performance bottlenecks and failure modes
• Understanding of hardware accelerators and networking topologies
• Experience optimizing ML data pipelines

Benefits

• Equity
• Medical, dental, and vision premiums fully paid for employees and dependents
• 401(k) matching up to 4% of base pay
• Unlimited PTO
• Company-wide Refill Days