Research Scientist, Open Benchmarks Red Team

Role overview Help make evaluations of advanced AI systems more reliable by finding weaknesses in benchmarks and working with their authors to validate and fix them. The work combines hands-on testing of models and agents, research into recurring evaluation failures, and engineering improvements that can be reused across tasks.

Responsibilities - Probe benchmarks for grading errors, ambiguous tasks, exploitable shortcuts, reward hacking, and environment vulnerabilities; reproduce issues so they can be investigated. - Propose repairs for confirmed problems, implement selected changes with benchmark authors, and check that fixes preserve valid solutions. - Build or adapt tools and methods that can detect recurring benchmark weaknesses across tasks and datasets. - Share findings with authors, coordinate improvements, and help prepare updated benchmark releases. - Contribute to technical reports, open-source tools, publications, and new approaches to evaluating benchmarks.

Requirements - Research or engineering experience in machine learning, large language models, AI agents, and evaluation. - Strong Python skills and experience building, running, and debugging agents or evaluation systems. - Ability to identify limitations in evaluation methods, develop alternatives, and test them rigorously. - Experience designing experiments, analyzing model behavior, and investigating unexpected failures. - Familiarity with agent or language-model benchmarks, evaluation frameworks, or automated graders. - Ability to take research or engineering work from problem discovery through implementation and validation, and communicate findings clearly.

Nice to have - Experience with adversarial testing, red teaming, reward hacking, or specification gaming. - Work on benchmarks for coding agents, computer use, or professional workflows; familiarity with reinforcement-learning environments, agent trajectories, verifiers, or evaluation infrastructure. - Publications or open-source work in AI evaluation, robustness, or agent systems. A relevant doctorate or equivalent research or industry experience is also valued.