AI Benchmark Engineer | Native Language Specialist - Spanish (Mexico) - Remote

Role overview A remote, freelance opportunity invites experienced software engineers to help construct a rigorous evaluation suite of Terminal-Bench tasks targeting large language models on multilingual software challenges. The mission centers on measuring multilingual robustness across prompt-language effects, non-English data handling, and locale or encoding edge cases in terminal workflows. Work is delivered as an independent contractor engagement, with flexible scheduling and project-based demand.

Responsibilities - Design realistic task environments using Spanish (Mexico) datasets and files that remain in the target language to genuinely stress multilingual handling. - Build reference implementations and write deterministic verifier scripts, falling back to rubric-based judging only when strictly necessary. - Probe for AI failure points when systems operate in native-language modes, including non-English data processing paths. - Calibrate task difficulty from Easy through Very Hard by analyzing execution logs across multiple model tiers using standard Terminal-Bench configurations. - Contribute to a four-layer human quality control process (creation, human review, calibration review, and audit) alongside automated LLM-based checks.

Requirements - 5+ years of industry experience in software engineering. - Background at leading technology organizations and/or a degree from a top-tier engineering university. - Native or near-native Spanish (Mexico) fluency with deep understanding of grammar, register, and phrasing rules, plus high English proficiency. - Strong command of Python, standard shell scripting, and data processing tooling. - Extensive Terminal/CLI-based development experience and familiarity with coding agents. - Deep technical grasp of multilingual text processing pitfalls: encoding and decoding robustness, Unicode normalization, locale-dependent conventions (collation, casing, non-Gregorian dates), text I/O, toolchain interoperability, and safe string operations.

Benefits and work setup - Independent contractor engagement with flexible scheduling and no fixed hours. - Competitive rates with prompt payment; suited as supplemental rather than primary income given fluctuating project demand. - 1099 status: no health insurance, paid time off, or retirement contributions; contractors handle their own tax obligations. - Geographic restrictions may apply in regions subject to international embargo or sanctions.