Research Scientist, Benchmarks & Evals

Role overview Build and evaluate benchmarks that show how real-world, opt-in image, video, and document data affects frontier AI models. You will take research from experimental design through model training, evaluation, and publication, then use the findings to guide future data collection and communicate results to research and business audiences.

Responsibilities - Design and release public benchmarks for image, video, and document models, alongside private test sets based on data not used in model training. - Compare models trained with and without particular data to measure its effect on performance. - Develop human evaluation studies and automatic metrics, using sound experimental controls and analysis. - Publish leaderboards and reports that researchers can use to assess model performance. - Identify capability gaps and recommend what data to collect next. - Share findings with research partners and prospective data buyers to inform data licensing discussions.

Requirements - At least three years of machine learning research or research engineering experience. - Evidence of building a benchmark or evaluation that others have used. - Hands-on experience training or fine-tuning image or video models with PyTorch. - Strong understanding of statistics and experimental design, including controls, ablations, and human studies. - Experience carrying experiments from GPU runs through analysis and final reporting. - Clear writing for both technical researchers and business audiences.

Nice to have - Research publications, particularly on datasets or benchmarks. - Experience on an AI lab’s evaluation or data team, licensing data to AI labs, or researching data value or attribution.

Benefits and work setup The research stack includes Python, PyTorch, open-weight image and video models, and cloud GPUs. The work draws on a large collection of opt-in photos, videos, and documents. The listing gives a base salary range of $150,000–$250,000 and an equity range of $350,000–$750,000.