Senior AI Product Manager, Evals & Annotations

Role overview

Own the product direction for tools that help teams evaluate AI applications and agents, collect human feedback, and improve system behavior over time. The role spans the evaluation lifecycle—from creating and running evaluations through review, analysis, and iteration—and calls for a product leader who can shape both technical strategy and user experience. You will work closely with customers, design, engineering, and senior product stakeholders to take ideas from investigation and prototyping through launch and refinement.

Responsibilities

- Set strategy and roadmap for evaluation, execution, review, and annotation capabilities. - Explore workflows and build prototypes, then partner with design and engineering to deliver polished, technically sound experiences. - Work through technical decisions involving evaluation execution, backend services, orchestration, and data infrastructure. - Speak with customers regularly to understand how they assess AI systems and validate product direction. - Prioritize customer needs, platform investments, and emerging opportunities in AI and agent development. - Guide work from problem definition through launch, adoption, and continued iteration.

Requirements

- Significant experience owning technically complex, developer-facing products; the listing prioritizes depth and product judgment over a fixed number of years. - Strong understanding of LLM evaluations and AI development workflows, including automated assessment, experimentation, and human feedback. - Experience with developer tools, AI infrastructure, machine learning platforms, or related technical products, and a sound grasp of software architecture. - Enough technical fluency to investigate issues in code, make small fixes, prototype ideas, and reason through implementation tradeoffs with engineers. - Strong product and user experience judgment, with the ability to define and prototype experiences in partnership with design. - Initiative, clear communication, and comfort making decisions in an ambiguous, fast-moving environment.

Nice to have

Experience with evaluation platforms, annotation or human-review workflows, agent orchestration, distributed systems, workflow orchestration, data infrastructure, or hands-on software development.

Benefits and work setup

The stated annual salary range is $200,000–$250,000, with equity and benefits that include health coverage, a 401(k), paid time off, parental leave, and mental wellness support. The company is remote-first, with optional offices in New York City and the San Francisco Bay Area; other employees can receive a monthly work-from-home stipend.