Evaluation Engineer Applied AI

You will discover priority use cases with customers and experts, design and score benchmarks, and build evaluations in real software. You will source or create data, run evaluations on models, identify flaws, collaborate with post-training teams, and present actionable results to leaders.

Responsibilities

• Run discovery with customers, industry experts, and internal teams to prioritize use cases
• Design benchmarks for industry use cases and frontier capabilities
• Define benchmark scoring methods
• Build and run evaluations where agents work in real software
• Maintain reliable evaluation results over time
• Stress-test evaluations for flaws and bugs
• Source data with expert vendors
• Build proxy or synthetic data when customer data cannot leave the customer
• Present findings to leaders
• Help teams build reusable evaluation tools and guides

Requirements

• At least 2 years of industry experience in software, AI, or consulting, or a PhD
• Software engineering or research background
• Experience writing code that others can build on
• Experience working directly with customers, industry experts, or other teams
• Experience building with LLMs
• Experience working with large datasets and cleaning them
• Strong Python skills
• Experience working in a large shared codebase and code review
• Understanding of LLM training, inference, and evaluation
• Fluency in English

Benefits

• Healthcare coverage
• Parental leave
• Retirement plans
• Relocation support
• Wellness programs
• Meal allowances
• Transportation allowances
• Location-specific perks