AI Benchmark Engineer | Native Language Specialist - Portuguese (Brazil) - Remote

Role overview This remote freelance role contributes to a multilingual evaluation suite that tests how well large language models handle software challenges in non-English contexts. The work centers on designing terminal-based benchmark tasks for Portuguese (Brazil), with assets deliberately kept in the target language to expose real localization failures. It is a contract, project-based opportunity suited to engineers who want flexible scheduling and supplemental income.

Responsibilities - Engineer benchmark tasks that evaluate coding agents on multilingual software challenges - Build realistic task environments using Portuguese-language datasets and files - Identify failure points where AI underperforms when prompted in the native language - Develop reference implementations and deterministic verifier scripts - Calibrate task difficulty across model tiers using standard terminal benchmark configurations - Participate in a four-layer human quality control process plus automated LLM checks

Requirements - 1+ years of industry experience in software or prompt engineering - Native or near-native Portuguese (Brazil) fluency with deep grammar and register awareness, plus high English proficiency - Strong proficiency in Python, standard shell scripting, and data processing - Extensive experience with terminal and CLI development workflows and familiarity with coding agents - Deep understanding of multilingual text processing pitfalls, including Unicode normalization and locale-dependent conventions

Nice to have - Experience with bidirectional text handling, font fallbacks, or rendering concerns specific to relevant scripts

Benefits and work setup - Independent contractor (1099) arrangement with no fixed hours or micromanagement - Competitive rates and prompt payment - Not intended as primary income; no health insurance, paid time off, or retirement benefits - Geographic restrictions may apply based on international sanctions - Contractors responsible for their own tax obligations