Remote job
Software Engineer, Evaluation Tooling
Job details
About this role
Role overview Build internal software that runs evaluations of AI assistants and language models at scale. The work covers querying systems, preserving the context needed to reproduce their outputs, and helping research and analysis teams compare results across models and versions.
Responsibilities - Build services that query AI assistants and language models through APIs and other interfaces. - Create data pipelines that store outputs with enough context to reproduce each run. - Develop tools for comparing answers across models, versions, and time. - Build internal interfaces for running, reviewing, and annotating evaluations. - Keep tools reliable, tested, secure, and well documented.
Requirements - Professional experience building production software in TypeScript or Python. - Experience with data pipelines, APIs, and relational databases such as PostgreSQL. - Familiarity with language model APIs and practical constraints such as rate limits, cost, and nondeterministic outputs. - Care for correctness, testing, and reproducibility.
Benefits and work setup - Full-time role based in Scottsdale, Arizona, or remote within the United States. - Annual pay range of $140,000–$220,000. - Health insurance and paid time off.