Staff GPU Inference SDET

You will build automated test frameworks, regression gates, and release qualification pipelines for GPU inference systems. You will validate distributed serving workloads, performance models, numerical correctness, resilience, and recovery; integrate observability and CI/CD; and investigate software and hardware failures.

Responsibilities

• Design automated test frameworks, regression gates, and release qualification pipelines
• Benchmark and stress-test distributed LLM serving frameworks
• Build workload replay and benchmarking tools
• Validate model accuracy, precision stability, determinism, and output correctness
• Engineer fault-injection and recovery tests for GPU clusters
• Integrate test pipelines with telemetry and continuous monitoring

Requirements

• Eight or more years of software engineering experience as an SDET, infrastructure quality lead, or systems test engineer
• Experience bringing up, provisioning, and validating multi-node GPU clusters
• Knowledge of LLM serving engines, prefill and decode disaggregation, KV-cache management, and dynamic batching
• Expert Python programming and test automation framework experience
• Kubernetes, Slurm, or Ray
• InfiniBand, RoCE, or NCCL
• Root-cause analysis, stress testing, and node failure simulation