Member of Technical Staff, LLM Evaluation Infra

You will build automated evaluation pipelines for model training and deployment. You will develop benchmarks and metrics, analyze model outputs for failures and biases, and turn real-world use cases into meaningful evaluation criteria.

Responsibilities

• Build scalable automated evaluation pipelines for model training and deployment
• Design and maintain evaluation frameworks and benchmarks for language models
• Analyze model outputs to identify failure modes, biases, and performance gaps
• Translate real-world use cases into evaluation criteria
• Define metrics for model quality, safety, reliability, and regression detection

Requirements

• Machine learning evaluation
• Applied machine learning research
• Git
• Docker
• AWS
• GCP
• Azure
• Large language model fundamentals
• Python
• PyTorch
• Evaluation metrics
• Generative model benchmarks
• Communication

Benefits

• Equity
• Flexible vacation and paid time off
• Health insurance
• Dental insurance
• Vision insurance
• 401k match
• Catered meals
• Commuter subsidies