Member of Technical Staff, ML Platform (Evals)

Role overview Build and lead the evaluation platform used to assess machine learning models across research and production. The work spans large-scale sample generation, automated scoring, human review, and clear reporting so research teams can compare experiments and make informed release decisions.

Responsibilities - Design the platform and tools for generating, annotating, reviewing, and adapting model evaluations at scale. - Build interfaces and storage systems across command-line tools, APIs, and browser-based workflows. - Partner with teams working on video, image, audio, agents, and robotics to understand their evaluation needs and turn solutions into shared capabilities. - Establish practices for reproducibility, metric definitions, reporting, and deciding when results are reliable enough to act on. - Drive adoption and integrate evaluation workflows into model training and serving. - Contribute to infrastructure that supports training and deployment of advanced models.

Requirements - At least five years building production machine learning infrastructure or data platforms, including work on evaluation, experimentation, or benchmarking. - Strong Python and PyTorch skills, plus hands-on experience running large GPU batch workloads on Kubernetes. - Experience building data pipelines and storage for large collections of media or model outputs, with versioning and reproducibility in mind. - Familiarity with experimental statistics, such as paired comparisons, confidence intervals, multiple-comparison risks, and inter-rater agreement. - Ability to build internal tools from command line through browser, set technical direction across teams, and drive a roadmap independently. - Understanding of the model development lifecycle and a practical approach to reliability and collaboration with research teams.

Nice to have - Experience creating evaluation suites for generative models, using language or vision-language models as judges, or connecting offline metrics to product experiments. - Experience evaluating AI agents or robotics policies.