Remote job
Staff Data Scientist, AI Evaluation
Job details
About this role
Role overview
Develop rigorous ways to measure and improve the quality and performance of AI systems. This hands-on data science role spans applied AI, engineering, and product analytics: you will work with production data, design experiments, investigate system behavior, and turn findings into decisions and improvements.
Responsibilities - Create quantitative evaluation methods and tools for assessing AI system quality and performance. - Design and run experiments that compare approaches, reveal tradeoffs, and measure the effects of changes. - Establish repeatable monitoring to detect regressions, anomalies, and shifts in system performance. - Extract, transform, and analyze production data for technical and product investigations. - Diagnose issues involving data, models, and production systems, then identify likely causes and possible remedies. - Partner with Engineering, Product, and Analytics to communicate findings and guide improvements.
Requirements - Strong data science skills for evaluating complex systems and interpreting results. - Experience designing experiments and quantitative measures of quality or performance. - Ability to work with production data, including extracting, transforming, and analyzing it. - Skill in investigating problems across data, models, and deployed systems. - Clear communication and ability to collaborate across technical and product teams.