Quality Evaluator – Fluent English (Colombia)

Role overview Evaluate responses from multiple AI models and help improve their accuracy, clarity, and usefulness. The work involves reviewing outputs across varied prompts, applying rating criteria, and explaining judgments in writing. This is a remote, part-time independent contract for up to three months, open to candidates in Colombia.

Responsibilities - Use different AI models and tools to review responses to a range of prompts. - Compare two or more responses and judge which best addresses the user’s needs. - Rate accuracy, helpfulness, instruction-following, clarity, and overall quality. - Apply detailed rubrics and multi-step workflows consistently. - Write concise, well-supported explanations for evaluation decisions.

Requirements - Fluent English, with strong reading comprehension and writing ability. - Extensive practical experience using large language models and assessing their outputs critically. - Sound analytical judgment across multiple dimensions of response quality. - Careful attention to detail and ability to follow complex guidance. - Ability to communicate reasoning clearly and switch comfortably between tools and workflows.

Nice to have - Experience in AI response evaluation, data annotation, or quality assurance. - Background in editing, research, fact-checking, or analytical writing. - Familiarity with rating rubrics and side-by-side comparisons.