Software Engineer Model Routing and Inference

You will build the inference platform that supports AI interactions in the product. You will evolve an inference gateway across provider APIs, design cross-provider failover, and develop routing backpressure and admission controls to prevent traffic spikes from affecting providers.

Responsibilities

• Build and evolve the inference gateway
• Design cross-provider failover
• Design routing backpressure and admission control

Requirements

• Experience building high-throughput low-latency distributed systems
• Experience with inference serving traffic routing or real-time data pipelines
• Ability to assess cost and performance tradeoffs at scale
• Software engineering fundamentals
• Experience shipping production systems that handle millions of requests
• Judgment regarding reliability cost latency and user experience