Sr. Staff Software Engineer, AI Platform

Role overview Set technical direction for a shared AI and machine learning evaluation platform used across a range of product and engineering teams. You’ll help make the platform reliable and broadly adopted, while shaping how teams assess model quality, safety, and readiness for release. The work spans architecture, evaluation practice, and coordination across engineering, research, product, and safety partners.

Responsibilities - Identify opportunities to modernize and scale a company-wide evaluation platform, and shape its roadmap with technical leaders. - Define technical strategy across teams, align partners, and drive adoption of shared evaluation capabilities. - Make architectural decisions that improve platform reliability and the experience of teams using it. - Establish evaluation approaches that combine automated measures with expert judgment, human feedback, safety considerations, and business outcomes. - Lead complex cross-team initiatives, resolving technical tradeoffs and risks through delivery.

Requirements - At least 3 years of hands-on experience designing and operating evaluation systems for generative AI and complex machine learning, including benchmarks, reference datasets, automated metrics, human review, model-based grading, and failure analysis. - Strong Python programming skills and a solid understanding of distributed systems. - Familiarity with evaluation frameworks for generative AI and recommendation systems, such as AutoEvals, OpenEvals, or Ragas. - At least 5 years building scalable platforms for experimentation, data and model versioning, orchestration, observability, release management, and production monitoring. - Sound judgment on AI risks, including robustness, bias, privacy, security, and the limits of automated signals. - A degree in computer science, engineering, or a related field, or equivalent experience.

Benefits and work setup The role can be based anywhere in the country, with in-person collaboration expected one to two times per quarter. Relocation assistance is not available.