Senior Inference Service SRE and Automation Engineer

You will own end-to-end reliability for customer inference endpoints, from API gateway to GPU model replicas and shared control-plane dependencies. You will define SLOs, manage capacity and incidents, automate remediation, run failover and resilience drills, improve observability, and protect production changes through safe rollout practices.

Responsibilities

• Own availability, latency, request success rate, and SLA delivery for inference endpoints
• Operate the API gateway, inference router, scheduler, queue, serving path, and KV cache service
• Manage inference runtime health, model validation, rollout, and rollback
• Design model-replica topology and failure isolation
• Manage multi-cluster traffic steering, failover, and regional recovery
• Monitor GPU fleet health and automate draining, cordoning, and replica replacement
• Operate Kubernetes and shared control-plane dependencies
• Engineer capacity, admission control, load shedding, and graceful degradation
• Define and instrument SLIs, SLOs, SLAs, error budgets, and alerting
• Maintain observability and external synthetic inference probes
• Lead incident management, customer communication, postmortems, and follow-up actions
• Implement staged rollout, canary deployment, and automatic rollback
• Run failover drills, disaster recovery tests, game days, and backup-restore verification
• Convert manual interventions into automated workflows

Requirements

• Have 5 or more years of SRE or production platform operations experience
• Have 2 or more years supporting a customer-facing API with a published SLA
• Have deep Kubernetes operations experience
• Have production experience serving LLM or model inference with vLLM, SGLang, TensorRT-LLM, Triton, or equivalent
• Have GPU fleet operations experience including DCGM, Xid, ECC, NVLink, and NCCL diagnosis
• Know traffic engineering, load balancing, gateways, health checks, DNS/GSLB, and failover
• Know SLI, SLO, SLA, error budgets, incident command, postmortems, and capacity planning
• Have experience operating PostgreSQL, Redis, message queues, object storage, and secrets management
• Know Terraform, Helm, GitOps, ArgoCD, or Flux
• Program in Go or Python for automation and operator development