Distributed LLM Inference Engineer

You will deliver end-to-end batch and online inference solutions at scale. You will integrate Ray Data and LLM engines, optimize cost and performance, work with open-source communities, contribute improvements, and apply current research and engineering practices to large-scale ML inference.

Responsibilities

• Ship end-to-end batch and online inference solutions at high scale
• Integrate Ray Data and LLM engines
• Optimize large-scale ML inference for cost and performance
• Integrate vLLM and contribute improvements to open-source software
• Follow and implement state-of-the-art open-source and research practices

Requirements

• Experience running large-scale ML inference with high throughput and low latency
• Knowledge of deep learning and deep learning frameworks such as PyTorch
• Understanding of distributed systems and ML inference challenges

Benefits

• Stock options
• Healthcare plans with 99% employer-covered premiums for employees and dependents
• 401k Retirement Plan
• Education & Wellbeing Stipend
• Paid Parental Leave
• Fertility Benefits
• Paid Time Off
• Commute reimbursement
• In-office meals