← Back to jobs

Remote job

AI QA Engineer, LLM Evaluation and Test Automation

AI Engineer Full-time Remote

Job details

Not specified Salary
Remote Eligibility
Not specified Experience
Full-time Employment

About this role

Role overview Build quality checks for AI features by combining automated testing with careful evaluation of language-model outputs. The work focuses on detecting failure modes, investigating their causes, and producing clear evidence about how well LLM-powered systems behave.

Responsibilities - Create and maintain automated tests for AI features and their surrounding software. - Evaluate model outputs against defined expectations, accounting for ways LLMs can fail. - Use UI automation to exercise application behavior and catch regressions. - Debug failures and investigate their underlying causes. - Write clear reports and judgments about generated text and test results.

Requirements - Experience with Python and a test framework such as pytest. - Experience with Playwright or a comparable UI automation tool. - Strong understanding of common LLM failure modes and how to test for them. - Strong debugging and root-cause analysis skills. - Excellent written English for evaluating generated text and documenting findings.

Nice to have - Experience red-teaming or security testing LLM applications. - Familiarity with human annotation workflows. - Knowledge of retrieval-augmented generation (RAG) or agent architectures.

Skills detected in the listing

PythonLLM
Detected Nov 15, 2026
Last verified Nov 16, 2026

Hidden Jobs Access

Unlock application links

Read the full job details for free. An active Hidden Jobs Access subscription is required to open the original application link.

Weekly

$6.99 /week
  • Original application links
  • Full access to all MCP features
  • Cancel anytime

Monthly

$35.99 $17.99 /month
  • Original application links
  • Full access to all MCP features
  • 35% cheaper than weekly

Lifetime

$99.99 $49.99 /forever
  • Original application links
  • Full access to all MCP features
  • One-time payment
Hidden Jobs gives subscribers direct access to original application links