Staff Software Engineer (AI Evals)

Role overview

Assess how AI coding assistants handle realistic software development tasks, with a focus on whether their responses show sound engineering judgment. This contract role centers on reviewing the full interaction—including explanations and guidance—rather than writing production code or checking syntax line by line.

Responsibilities

- Review AI-generated coding interactions from start to finish. - Judge whether responses are useful, broadly correct, and consistent with how an experienced engineer would approach the task. - Evaluate the clarity and value of explanations, preambles, and reasoning alongside code output. - Distinguish between levels of response quality and explain what separates an adequate answer from a strong one. - Give direct feedback on what worked, what was misleading, and what could undermine a developer’s trust. - Help establish standards for high-quality interactions with AI development tools.

Requirements

- Staff- or principal-level engineering experience, or equivalent expertise. - Strong background in at least one of Go, Python, Rust, TypeScript, or JavaScript. - Hands-on experience with AI coding tools such as Codex, Claude Code, or Cursor, plus familiarity with AI-assisted development workflows. - Ability to assess code and engineering decisions without executing every suggestion or reviewing every line in depth. - Confidence making rigorous, subjective judgments and giving clear, opinionated feedback. - High standards for engineering quality and for how an experienced developer should be guided.

Nice to have

- Experience with AI-first IDEs, prompt design, or evaluation processes. - Experience mentoring senior engineers or setting engineering standards.

Benefits and work setup

- Contract engagement at $100–$200 per hour, with approximately 20–40 hours per week and a start as soon as possible. - Expected to run through October, with possible extensions. - Selection includes a take-home evaluation exercise and a background check.