Senior QA Engineer AI

You will create and maintain evaluation datasets, design scoring methods, and run evaluations for release candidates. You will operate release gates, identify AI failure modes, investigate production traces and tool calls, turn findings into regression coverage, and define and improve quality metrics for AI product surfaces.

Responsibilities

• Build and maintain verified golden datasets for AI products
• Design deterministic and model-graded scoring for AI outputs
• Schedule and execute evaluations for release candidates
• Compare evaluation results with baselines and triage failures
• Operate the release gate for prompt and model changes
• Identify AI failure modes, including hallucination, incorrect tool selection, context loss, and PII exposure
• Investigate production traces to determine retrieval, generation, and tool failures
• Convert findings into engineering evidence and regression coverage
• Define, report, and improve AI quality metrics

Requirements

• At least 1 year owning quality for AI products in a lead or primary-owner capacity
• Hands-on experience building or operating an AI evaluation harness
• 4–6 years of QA or SDET experience
• Working proficiency in Python
• Ability to analyse traces and tool calls using observability tooling
• Strong API testing experience
• Ability to build tooling and automation
• Clear written communication