Software Engineer

Role overview Build and operate software that supports large-scale evaluation of coding agents and language models. The work spans backend services, data models, APIs, background processing, and user-facing interfaces, with a focus on producing dependable benchmark results and quality checks. This role suits an experienced engineer who can take loosely defined needs through design, implementation, and release.

Responsibilities - Deliver features across data storage, APIs, background workers, and frontend screens as needed. - Run and improve evaluations in isolated environments, and turn many trial results into useful comparisons. - Develop checks and grading approaches to assess whether benchmark tasks are clear, solvable, and scored fairly. - Add support for models, agents, and benchmarks using reusable designs. - Maintain performance and reliability through efficient database queries, safe migrations, and robust concurrency. - Review code changes and help uphold shared engineering practices.

Requirements - At least five years building and operating production backend systems, including ownership of a service from design through on-call support. - A degree in computer science, software engineering, or a related field. - Strong Python skills and experience with Django or a comparable framework, PostgreSQL, and a task queue such as Celery. - Practical experience with LLM evaluation, including benchmark design, agent trial scoring, and aggregating and comparing results. - Sound judgment in identifying unclear tasks, weak or exploitable verifiers, answer leakage, and misleading checks. - Ability to own ambiguous work through implementation, migrations, tests, and edge cases; comfortable with TypeScript and React when frontend changes are needed.

Nice to have - Familiarity with evaluation harnesses such as Harbor, Terminal-Bench, SWE-bench, or Inspect. - Experience running agents in containers or cloud sandboxes, or with rubric-based grading and reducing judge bias. - Knowledge of coding agents, MCP, distributed scheduling, concurrency limits, or database locking at scale. - Contributions to open-source evaluation or infrastructure projects.

Benefits and work setup The listing describes work at the intersection of AI evaluation and research, with opportunities to contribute to benchmark and evaluation infrastructure. Compensation is described as base salary plus equity, but the published salary figures are incomplete.