Principal Infrastructure Engineer AI Cluster Performance and Validation

You will define cluster acceptance criteria, run AI workloads at scale, diagnose performance failures, and build automated validation systems. You will optimize GPU, network, storage, scheduler, and host configurations, create Python tooling, and establish reliability and benchmarking standards.

Responsibilities

• Define architecture, roadmaps, acceptance criteria, and performance standards for GPU clusters
• Run distributed training and inference workloads to validate cluster behavior
• Diagnose cluster failures and performance regressions across the full stack
• Design and build automated validation and burn-in systems
• Optimize fabric, communication, storage, and host-level configurations
• Translate performance needs into engineering solutions with partner teams
• Build Python systems for automated triage, telemetry correlation, and regression detection
• Establish standards for reliability, observability, benchmarking, and operational excellence
• Mentor engineers and produce runbooks and post-incident write-ups

Requirements

• 10+ years of relevant large-scale compute infrastructure experience
• AI workload experience at scale
• Distributed training
• PyTorch, Megatron-LM, DeepSpeed, or equivalent
• Cluster validation
• Performance debugging
• InfiniBand or RoCEv2
• RDMA
• GPUDirect
• TCP/IP
• BGP
• Linux systems
• Python
• C, C++, or Go
• Ansible or Terraform
• SLURM or Kubernetes