GPU Cluster Infrastructure Engineer

You will review GPU cluster designs and bills of materials, lead hardware acceptance testing, bring up InfiniBand fabric, and validate high-performance storage. You will build secure out-of-band management and firmware baselines, integrate telemetry and health checks, document operational procedures, and provide post-launch escalation support.

Responsibilities

• Review cluster designs and bills of materials across compute, networking, and storage
• Lead hardware acceptance testing, including cabling, optics, InfiniBand fabric bring-up, burn-in, and vendor deliverables
• Stand up and validate high-performance storage with vendor teams
• Build out-of-band management and firmware baselines and secure the management plane
• Integrate hardware, fabric, and storage telemetry into the observability stack
• Create alerting and automated health checks
• Write runbooks, as-builts, and remote-hands procedures
• Provide escalation support after go-live and help ramp up the team

Requirements

• Build and operate NVIDIA HGX or DGX clusters in production at a GPU cloud, HPC center, or AI lab
• Have hands-on InfiniBand experience, including subnet management, UFM, fabric bring-up, and degraded-link and optics diagnosis
• Have experience with NDR or newer InfiniBand
• Perform GPU node bring-up and burn-in, including firmware, BMC/Redfish, DCGM, NCCL testing, PXE, imaging, and XID error triage
• Have parallel storage experience with WEKA, VAST, GPFS, Lustre, or similar
• Work effectively both on the data center floor and remotely, including directing colo remote hands
• Troubleshoot across hardware, fabric, and software and communicate clearly
• Be a US citizen or hold a US visa

Benefits

• Meaningful equity
• Health, dental, and vision coverage with 90% coverage for employees and 50% for dependents
• Opportunities to participate in cloud native community events
• Fitness stipend
• Learning budget