Director of Evaluations

You will define and lead evaluation strategy, build an independent evaluations function, and develop benchmarks, datasets, red-teaming programmes, and scalable tooling. You will evaluate AI capabilities, safety, explainability, and adversarial robustness, communicate results publicly, and represent evaluation work externally.

Responsibilities

• Define the evaluations strategy and roadmap
• Build and scale the Evaluations Team
• Maintain independence from research and product streams
• Design benchmarks, datasets, tasks, and interactive environments
• Evaluate capabilities, safety, explainability, and adversarial robustness
• Lead guardrail evaluations for frontier models
• Establish automated and manual red-teaming programmes
• Build evaluation tooling and infrastructure
• Publish evaluation results through reports, system cards, publications, and blog posts
• Represent evaluation work with external organizations

Requirements

• Advanced degree in machine learning, computer science, or a closely related field
• 10+ years of machine learning experience
• At least 5 years of technical team leadership experience
• Expertise designing and running large-scale evaluations of LLMs or frontier ML systems
• Experience building evaluation datasets, benchmarks, or interactive environments
• Written and verbal communication skills

Benefits

• Comprehensive health benefits
• Minimum 20 days of vacation per year upon start
• Minimum 4% employer retirement savings contribution
• Flexible benefits for well-being