Remote job
Member of Technical Staff, Evals
Job details
About this role
Role overview Develop evaluation methods that help teams understand the capabilities and performance of machine learning models. The work spans model training and deployment, combining benchmark design, dataset analysis, and careful investigation of whether results reflect the behaviors they claim to measure. Accuracy, reproducibility, and clear interpretation are central to the role.
Responsibilities - Design and validate evaluations and benchmarks for models across training and deployment. - Investigate model behavior and identify areas that current evaluation methods miss. - Analyze dataset quality and run data ablations to find opportunities to improve training data. - Audit benchmarks, research papers, and open-source evaluation implementations for errors or flawed assumptions. - Define metrics that meaningfully reflect model quality and improve the reliability of measurements. - Build tools that make evaluation results accessible and useful to teams working on models and products.
Requirements - Experience developing model evaluations, benchmarks, or machine learning experiments. - Strong programming skills and experience working with machine learning systems. - A record of owning technical projects from planning through completion. - Strong understanding of post-training techniques and how they affect model performance. - Experience analyzing results, with a skeptical and evidence-based approach to findings that cannot be reproduced or explained. - Comfort with ambiguity and a commitment to rigorous, reliable evaluation.
Benefits and work setup The role can be based in San Francisco or remotely. The listed US annual salary range is $225,000–$550,000, depending on experience, with equity as a significant part of compensation. Benefits include a 401(k) plan with 6% salary matching, health, dental, and vision coverage for employees and dependents, and unlimited paid time off. Visa sponsorship and relocation support are available for candidates moving to San Francisco.