Lead Engineer, AI Platform

Role overview Lead the engineering work that measures and improves the quality, reliability, speed, and cost of production AI agents. This is a hands-on player-coach position: you will build evaluation and observability systems while leading an engineering team and shaping its roadmap. The focus is production infrastructure and diagnostics rather than model research or training.

Responsibilities - Build evaluation infrastructure, including CI/CD pipelines, scoring methods, and datasets for individual agent actions and multi-turn conversations. - Trace failures across agent planning, execution, tool selection, and tool use; turn findings into prototypes and fixes. - Develop observability and diagnostic tools that help teams understand agent behavior and performance. - Establish processes for measuring quality and tracking improvements in reliability, latency, and cost. - Lead and manage the AI quality engineering team while contributing directly to implementation. - Collaborate asynchronously across time zones and document decisions and findings.

Requirements - Strong software engineering experience and a record of shipping production systems with measurable results. - Comfortable working with Ruby on Rails or Python, with the ability to learn the other as needed; Python is useful for evaluation and data work. - Experience building evaluation or observability infrastructure for AI or ML systems, such as evaluation pipelines, scorers, dashboards, or CI/CD workflows. - Experience creating datasets, annotation processes, or labeling pipelines for trustworthy AI evaluation. - Ability to run frequent experiments, learn from unsuccessful approaches, and work through ambiguity in a fast-moving environment. - Strong spoken and written English proficiency, specified at CEFR C2 or ILR Level 5.

Nice to have Familiarity with evaluation frameworks such as Braintrust or LangSmith.

Benefits and work setup The organization is remote-only, with a highly asynchronous, documented working style and team retreats twice a year. Compensation is benchmarked to U.S. levels globally, with equity and ongoing refresh grants, 35 days of paid time off annually, and benefits supporting health, wellbeing, and professional development.