Expert: Software Engineer – AI Evaluation (Task-based Contract, Remote)

Role overview Create and assess practical engineering exercises that reveal how well AI models handle coding, technical reasoning, and detailed instructions. You’ll draw on professional software development judgment to build realistic scenarios, define what good work looks like, and explain where model responses succeed or fall short.

Responsibilities - Design original, multi-step tasks using fictional codebases, technical specifications, architecture documents, or datasets. - Review generated code and technical content for correctness, efficiency, clarity, and sound engineering practice. - Score responses against reference answers and rubrics, providing specific, structured feedback. - Refine tasks to reflect real engineering complexity and review other contributors’ work for accuracy and practical relevance. - Work independently and asynchronously, meet deadlines, and incorporate reviewer feedback.

Requirements - At least two years of professional software engineering experience; internships do not count. - Strong coding ability in one or more major languages, such as Python, Java, C++, JavaScript, TypeScript, or Go. - Experience with production software, including building, reviewing, or debugging code. - Familiarity with system design, data pipelines, machine learning, or applications built with large language models. - Clear technical writing and the ability to evaluate work consistently against explicit criteria.

Nice to have - Background in code review, technical writing, AI training, or model evaluation. - Experience creating realistic engineering tasks and assessing their difficulty.

Benefits and work setup - Remote, task-based contract open to applicants in India; this is not a full-time position. - Flexible schedule, typically around 15–20 hours per week, with work that may fit alongside a full-time role. - Pay is listed at $50–$100 per hour, with some tasks paid at a flat rate. - Work must be original and human-authored. Use fictional scenarios and do not include employer or client materials, NDA-protected content, or copied published work.