Remote job
AI QA Engineer, LLM Evaluation and Test Automation
Job details
About this role
Role overview Build quality checks for AI features by combining automated testing with careful evaluation of language-model outputs. The work focuses on detecting failure modes, investigating their causes, and producing clear evidence about how well LLM-powered systems behave.
Responsibilities - Create and maintain automated tests for AI features and their surrounding software. - Evaluate model outputs against defined expectations, accounting for ways LLMs can fail. - Use UI automation to exercise application behavior and catch regressions. - Debug failures and investigate their underlying causes. - Write clear reports and judgments about generated text and test results.
Requirements - Experience with Python and a test framework such as pytest. - Experience with Playwright or a comparable UI automation tool. - Strong understanding of common LLM failure modes and how to test for them. - Strong debugging and root-cause analysis skills. - Excellent written English for evaluating generated text and documenting findings.
Nice to have - Experience red-teaming or security testing LLM applications. - Familiarity with human annotation workflows. - Knowledge of retrieval-augmented generation (RAG) or agent architectures.