Senior Software Engineer, AI Evaluation Infrastructure

Role overview Build and operate the software that helps teams assess coding agents and language models at scale. The work spans backend services, data models, background processing, and user interfaces, with a focus on producing reliable evaluation results and trustworthy benchmark tasks.

Responsibilities - Deliver features across APIs, data models, background workers, and UI as needed. - Run model and agent evaluations in sandboxed environments, then score and aggregate results. - Improve benchmark quality checks, including automated verifiers and rubric or model-based grading. - Integrate new models, agents, and benchmarks through reusable designs. - Maintain performance and reliability with efficient queries, safe migrations, and sound concurrency controls. - Review code changes and help uphold engineering standards.

Requirements - Around five or more years building and operating production backend systems, including ownership of a service through its operational lifecycle. - Strong Python skills and experience with Django or a similar framework, PostgreSQL, and a task queue. - Practical understanding of LLM evaluation: benchmark design, trial scoring, and comparison of aggregated results. - Ability to identify unclear tasks, weak or exploitable checks, answer leakage, and misleading scores. - Sound engineering judgment, with experience taking loosely defined work through implementation, testing, migrations, and edge cases. - Comfortable contributing to a TypeScript and React interface when needed, and communicating decisions clearly in reviews and design notes.

Nice to have - Experience with evaluation harnesses, such as those used for coding benchmarks. - Experience running agents in containers or sandboxed cloud environments. - Familiarity with judge-based grading, reducing scoring bias, distributed job scheduling, or database locking at scale. - Exposure to coding agents or the Model Context Protocol.