Software Engineer Inference

You will integrate model architectures into the inference engine and optimize deployments across clusters and hardware providers. You will build tools for inference jobs, automate and maintain reliable services, develop scheduling systems, and maintain CI/CD for model checkpoints and SDKs.

Responsibilities

• Integrate new model architectures into the inference engine
• Optimize model efficiency and deployments
• Build tooling to measure, profile, and track inference jobs and workflows
• Automate, test, and maintain inference services
• Manage and optimize inference workloads across clusters and hardware providers
• Build scheduling systems for GPU resources while meeting SLOs
• Maintain CI/CD for model checkpoints and SDKs

Requirements

• Python and system-architecture skills
• Experience deploying models with PyTorch, Hugging Face, vLLM, SGLang, TensorRT-LLM, or similar tools
• Experience with queues, scheduling, traffic control, and fleet management at scale
• Experience with Linux, Docker, Kubernetes, orchestration, deployment, and scheduling
• Familiarity with Redis and S3-compatible storage