Senior Python Engineer - LLM Code Evaluation

Role overview

This contract position centers on judging the real-world quality of AI-assisted coding workflows rather than shipping production software. The work involves reviewing how AI coding agents respond, reason, and explain themselves, then translating senior-engineer judgment into clear, opinionated feedback that helps shape evaluation standards. It is a part-time engagement suited to a Staff or Principal-level engineer who treats engineering taste as a serious craft.

Responsibilities

- Review end-to-end interactions with AI coding agents and assess whether the output reflects sound engineering judgment. - Judge the usefulness, plausibility, and alignment of generated responses with how a strong developer would think and respond. - Evaluate the quality of preambles, reasoning, and explanations, not only the code itself. - Calibrate quality across response tiers and articulate what separates strong output from mediocre output. - Provide direct, opinionated written feedback describing what works, what fails, and what feels misleading or off-target. - Contribute to a shared definition of what great looks like when developers collaborate with AI-first coding tools.

Requirements

- Staff, Principal, or equivalent senior engineering experience. - Strong background in Python, TypeScript, or JavaScript. - Hands-on familiarity with OpenAI Codex, Claude Code, Cursor, or comparable AI-assisted development tools. - Deep understanding of modern AI-assisted developer workflows and how they are typically used in practice. - Ability to evaluate code and interactions at a high level without executing or line-by-line reviewing every snippet. - Comfort making subjective but rigorous judgments and delivering candid feedback in writing.

Nice to have

- Prior exposure to prompt design, evaluation rubrics, or quality-rating workflows. - Experience mentoring senior engineers or setting engineering standards. - Background working inside AI-first IDE environments.

Benefits and work setup

- Hourly contractor engagement at $100–$200/hour. - Approximately 10–20 hours per week. - Initial term running through early May, with the possibility of extension. - ASAP start. - Hiring process includes a take-home evaluation exercise and a single behavioral interview.