Staff Technical Product Manager, Evals

Role overview Own the evaluation strategy for an AI coding agent at senior or staff product manager level. The central goal is to improve the agent’s speed, cost, and accuracy by building offline and online evaluations informed by product usage while protecting customer privacy.

Responsibilities - Define and maintain an offline evaluation benchmark and online evaluation program. - Use product activity and code repositories to identify useful evaluation cases. - Anonymize data used in evaluation work to protect customer privacy. - Partner with engineering on complex implementation, drawing on knowledge of system design, architecture, data modeling, and API design. - Apply evaluation findings to improve agent quality, performance, and cost.

Requirements - Prior experience creating or managing technical evaluations and familiarity with multiple evaluation methods and reward modeling. - Strong software fundamentals and the ability to code. - Product judgment grounded in an understanding of customer goals. - A record of shaping clear, usable product experiences. - Significant hands-on experience with AI agent context and harness practices, such as prompting, context management, summarization, memory, tools, and subagents. - Ability to work effectively with current AI coding tools.

Nice to have - Experience with TypeScript, React, agent frameworks, or evaluation platforms.

Benefits and work setup - Strong preference for working from an office in San Francisco; remote may be considered for exceptional candidates working US time zones and traveling to the office regularly. - Benefits include healthcare, vision and dental coverage, flexible vacation, and an AI usage budget. - Compensation is listed as $200,000–$350,000 plus equity, varying by seniority and location.