AI Evaluation Infrastructure Engineer

Role overview An infrastructure engineering role dedicated to building the tooling that makes high-quality AI evaluation possible at scale. The work spans execution engines, grader infrastructure, leaderboards, and feedback loops that connect offline scores to real user outcomes. The role supports multiple product teams shipping AI features faster and with greater confidence in the signal behind their decisions.

Responsibilities - Build an execution engine that scores candidate versions against task sets in minutes rather than hours - Create tooling that samples from production logs and validates tasks before they enter an evaluation set - Build grader infrastructure spanning ground-truth checks, rubrics, and LLM-as-judge approaches - Develop tooling for calibrating human reviewers, measuring judge-to-human agreement, and monitoring drift - Build leaderboards and reporting with sample size, confidence intervals, and run-to-run variance so teams can separate real gains from noise - Support in-product side-by-side serving, feedback capture, and implicit signal extraction from real conversations - Build the loop that compares offline scores to online outcomes and flags evaluation sets that have stopped predicting reality - Partner with product, engineering, data, and ML teams to make evaluation fast and trustworthy enough for daily development

Requirements - Experience building production platforms or infrastructure, including distributed batch execution or data pipelines that process production logs - Strong statistical literacy, including comfort with confidence intervals, variance, power, and multiple comparisons - Experience evaluating LLM or ML systems, or deep systems engineering experience with strong interest in AI evaluation - Product instinct for internal tools, with the understanding that leaderboards and annotation workflows only matter if teams use them - Bias toward building reliable, observable systems that other engineers can trust - Strong collaboration skills for working across ambiguous product, data, and engineering problems

Benefits and work setup - Base salary range of $263,600–$395,400 USD, with pay varying by location and market conditions - Remote work available across locations where the organization operates - Medical insurance, flexible time off, retirement savings plans, and modern family planning benefits