AI Benchmark Engineer | Native Language Specialist - German (Austria) - Remote

Role overview A remote, freelance position focused on designing and validating evaluation benchmarks that stress-test large language models on multilingual software tasks in German (Austria). The work targets prompt-language effects, non-English data handling, and locale/encoding edge cases inside terminal workflows, with the goal of producing high-signal tasks that genuinely measure multilingual robustness rather than English-translation workarounds.

Responsibilities - Engineer Terminal-Bench tasks that probe how coding agents handle German-language inputs, locale conventions, and encoding subtleties. - Build realistic task environments whose datasets, files, and artifacts remain in the target language to surface real multilingual failure modes. - Identify prompting and translation failure points where current models degrade in German, then craft tasks that expose them. - Develop reference implementations and write deterministic verifier scripts, using rubric-based judging only when strictly necessary. - Run tasks against multiple model tiers, analyze execution logs, and calibrate difficulty from Easy through Very Hard. - Contribute to a multi-layer quality control process combining human review, calibration review, audit, and automated model-based checks.

Requirements - Five or more years of industry software engineering experience. - Background at leading technology companies and/or graduation from a top-tier engineering university. - Native or near-native German (Austria) fluency with deep understanding of grammar, register, and phrasing, plus high English proficiency. - Strong proficiency in Python, standard shell scripting, and data processing. - Extensive experience with Terminal/CLI-based development workflows and working familiarity with coding agents. - Deep technical grasp of multilingual text-processing pitfalls, including Unicode normalization, locale-dependent collation and casing, non-Gregorian date formats, safe string operations, and toolchain I/O interoperability.

Nice to have - Familiarity with bidirectional/RTL handling, font fallbacks, and typography considerations for UI or generated artifacts.

Benefits and work setup - Fully remote independent-contractor engagement with flexible hours and no fixed schedule. - Competitive contractor rates with prompt payment and no invoice chasing. - Best treated as supplemental income rather than a primary role, since project availability fluctuates. - Access to a global community of linguists and subject-matter experts working on applied AI and language technology. - Geographic restrictions may apply based on international sanctions or embargoes; contractors handle their own tax obligations.