Software Engineer, Inference

You will integrate model architectures into the inference engine and scale deployments across thousands of machines. You will build tooling for inference workflows, automate and maintain reliable services, optimize workloads across clusters and hardware providers, and maintain scheduling and CI/CD systems for models and SDKs.

Responsibilities

• Integrate new model architectures into the inference engine
• Optimize model efficiency and deployments with research, engineering, and infrastructure partners
• Build tooling to measure, profile, and track inference jobs and workflows
• Automate, test, and maintain inference services for uptime and reliability
• Manage and optimize inference workloads across clusters and hardware providers
• Build GPU scheduling systems and maintain CI/CD for model checkpoints and SDKs

Requirements

• Strong Python and system-architecture skills
• Experience deploying models with PyTorch, Hugging Face, vLLM, SGLang, TensorRT-LLM, or similar tools
• Experience with queues, scheduling, traffic control, and fleet management at scale
• Experience with Linux, Docker, Kubernetes, orchestration, deployment, and scheduling
• Familiarity with Redis and S3-compatible storage