Lead Engineer, AI Quality

Role overview

A hands-on lead engineering role focused on the quality infrastructure behind production AI systems for a digital community and commerce platform. The position blends player-coach leadership with deep individual contribution across evaluation frameworks, observability tooling, and diagnostic pipelines for AI agents. The team owns the foundation for measuring, diagnosing, and improving how AI features behave at scale.

Responsibilities

- Design and own evaluation infrastructure, including CI/CD pipelines, scorers, and datasets that judge whether AI agents are working correctly from a single tool call through full multi-turn conversations. - Diagnose quality breakdowns by tracing failures across the agent pipeline, covering planning versus execution, tool selection, and tool trajectories inside complex product surfaces. - Build observability tooling and dashboards that surface quality signals alongside latency and cost-efficiency tradeoffs. - Turn diagnostic findings into prototypes that resolve issues or hand concrete recommendations back to product engineering. - Lead, mentor, and grow the AI Quality engineering team while driving an ambitious technical roadmap.

Requirements

- Significant engineering experience with a track record of shipping reliable production systems and proving impact through metrics and iteration. - Proficiency in Ruby on Rails, the production stack, with willingness to ramp up fast; strong Python skills are a plus for the evaluation and data side of the work. - Hands-on experience building evaluation or observability infrastructure for ML or AI systems, including eval pipelines, scorers, dashboards, or CI/CD for evals, with familiarity using frameworks such as Braintrust or LangSmith. - Experience designing datasets, annotation workflows, or labeling pipelines for AI evaluation, drawing on human annotation, AI-generated data, or simulation. - Comfort running many experiments, learning quickly, and operating in a fast-paced, ambiguous environment. - Native or near-native English proficiency at CEFR Level C2 or ILR Level 5.

Benefits and work setup

- Fully remote team with twice-yearly in-person company retreats in varied international destinations. - U.S.-benchmarked compensation offered globally, plus company equity with ongoing refresh grants. - 35 days of paid time off each year. - Asynchronous, documentation-first culture with high autonomy, flexibility, and a benefits package supporting health, wellbeing, and professional growth.