Senior AI QA Automation Engineer

Role overview

Build quality practices for AI agents and other AI-enabled systems, combining established software test automation with evaluations tailored to prompts, retrieval-augmented generation (RAG), and model behavior. The work spans development and production: define measurable release criteria, automate testing across interfaces and services, and monitor quality after deployment.

Responsibilities

- Create evaluation frameworks with benchmark datasets, reference responses, regression suites, metrics, and release thresholds for AI systems and workflows. - Test for hallucinations, grounding, citation accuracy, prompt injection, tool selection, and coordination among multiple agents. - Build UI, API, integration, performance, regression, and end-to-end automation. - Develop agent-assisted approaches for test-case generation, workflow and API validation, and user acceptance testing. - Add quality gates to CI/CD pipelines and monitor production reliability, quality changes, and recurring failures. - Translate business workflows and AI behaviors into measurable quality criteria.

Requirements

- At least five years in software quality, test automation, or reliability engineering. - Strong Python skills and experience building reusable test frameworks. - Hands-on experience with Playwright and/or Selenium, plus API and UI automation. - Experience with integration, regression, end-to-end, and CI/CD-based testing. - Experience evaluating RAG systems, prompts, and AI behavior, including with Azure AI Foundry Evaluations. - Ability to define evaluation metrics, benchmark data, regression suites, and quality thresholds. - Bachelor’s degree in computer science, software engineering, information systems, or a related field.

Nice to have

- Experience testing generative AI, copilots, or AI agents; using LLM evaluation frameworks or advanced prompt tests; or building agent-assisted QA tools. - Background in adversarial or Responsible AI testing, or in monitoring AI drift and failure patterns in production.

Benefits and work setup

- Remote role open to candidates in Costa Rica, Peru, Colombia, or Bolivia.