AI Benchmark Engineer | Native Language Specialist - French (France) - Remote

Role overview Contribute to a multilingual evaluation suite for large language models, designing and validating Terminal-Bench tasks that probe how well AI coding agents handle non-English software workflows. This is a remote, freelance engagement aimed at surfacing failure points around prompt language effects, locale-sensitive data processing, and tricky encoding edge cases in terminal environments.

Responsibilities - Engineer evaluation tasks that meaningfully stress multilingual handling without leaning on English translations. - Assemble realistic task environments using datasets and files written and structured in your native language. - Identify failure modes by probing how AI systems respond when prompted and translated in your native language. - Build reference solutions and write deterministic verifier scripts, reserving rubric-based judging for cases where it is unavoidable. - Run benchmarks across model tiers (Haiku, Sonnet, Opus) and calibrate difficulty from Easy to Very Hard based on execution logs. - Participate in a four-layer human quality control process alongside automated LLM-based checks to safeguard fairness and linguistic accuracy.

Requirements - 5+ years of professional software engineering experience, ideally with a track record at leading technology companies or a top-tier engineering university background. - Native or near-native fluency in French (France variant) with strong command of grammar, register, and phrasing, plus high English proficiency. - Strong proficiency in Python, standard shell scripting, and data processing. - Extensive terminal/CLI-based development workflow experience and familiarity with AI coding agents. - Deep understanding of multilingual text processing pitfalls, including Unicode normalization, encoding robustness, locale conventions (collation, casing, non-Gregorian dates), and safe string operations.

Nice to have - Experience with bidirectional/RTL handling, font fallback strategies, or typography considerations in UIs and artifacts where relevant to the target language.

Benefits and work setup - Independent contractor arrangement with flexible hours and no fixed schedule. - Remote work with project-based demand that may fluctuate; intended as supplemental rather than primary income. - No employer-provided health insurance, paid time off, or retirement contributions; contractors handle their own taxes. - Geographic restrictions may apply in regions subject to international embargoes or sanctions.