AI Agent Evaluator

Role overview Evaluate how well AI assistants handle personalized conversations by comparing two assistant versions in realistic, multi-turn interactions. Your assessments will focus on whether each assistant uses relevant context from connected digital services, responds naturally and safely, and adapts when you offer feedback. This is a remote, part-time independent contractor project expected to last up to three months.

Responsibilities - Write prompts based on your own questions, interests, or needs. - Carry out natural conversations with two AI assistants over multiple turns. - When relevant, ask questions that draw on information or activity from connected services. - Compare how accurately and effectively the assistants use relevant context. - Assess helpfulness, naturalness, trustworthiness, and safety. - Check how assistants respond to corrections and explain the reasoning behind your evaluations.

Requirements - Fluency in a supported language and locale: Latin American Spanish, Arabic, Brazilian Portuguese, Japanese, Turkish, Indonesian, Hindi, Korean, French, or German. - Strong reading comprehension, clear written feedback, attention to detail, and sound judgment. - Willingness to use relevant personal context and connected services during evaluation. - Recent activity across a qualifying combination of digital services and successful completion of eligibility verification. Meeting the activity threshold is required, and verification does not guarantee placement.

Nice to have - Familiarity with AI assistants for everyday questions, planning, or research. - Experience in AI evaluation, user research, or quality assurance. - Skill at noticing subtle differences in relevance, personalization, and conversation quality.