Multilingual Audio Evaluator

Role overview Evaluate how conversational AI agents communicate in real-time scenarios across supported languages. This project-based independent contractor work combines spoken-language roleplay with transcript labeling, technical session capture, and rubric-based assessment to help improve the quality and reliability of AI interactions.

Responsibilities - Check required tools and access before each live evaluation session. - Conduct real-time roleplay conversations with AI agents. - Capture recordings, session identifiers, and relevant tool-call or console logs. - Label transcripts according to project guidance and score interactions against structured rubrics. - Explain evaluation scores clearly in writing. - Inspect and troubleshoot JSON, identifying structural or formatting problems.

Requirements - C1 or C2 spoken fluency in at least one target language: Welsh, Dutch, Greek, Italian, French, German, or Portuguese. - Willingness to demonstrate speaking and listening ability through an assessment. - Strong JSON skills, including reading, validating, and diagnosing structural or formatting errors. - Comfort with digital tools and collecting recordings and technical logs. - Careful, consistent application of evaluation guidelines, with sound judgment and clear written communication.

Nice to have - Experience in AI evaluation, audio annotation, transcript labeling, technical support, quality assurance, or software development. - Familiarity with conversational AI, APIs, or structured data.

Benefits and work setup - Remote work may be available globally, subject to project location restrictions. - Project-based contract expected to last two to three months. Commitment and compensation were not specified.