QA Engineer

Role overview Establish quality controls for AI systems running in production. This role focuses on evaluating variable model outputs and connected workflows, keeping tests relevant to real usage, and determining whether a system is ready for delivery. It also tracks reliability, business claims, and gradual changes in quality or cost.

Responsibilities - Build and maintain evaluation sets for AI systems, updating them as real user traffic reveals new questions and edge cases. - Create regression checks for workflow changes, including downstream failures caused by altered outputs or formats. - Review business scenarios and client edge cases, assess whether results are usable, and make the release readiness decision. - Design a repeatable method to verify reported business outcomes on individual projects and flag claims that do not hold. - Monitor for quality drift and rising token costs, and create signals that surface changes early. - Write concise acceptance reports, defect notes, and drift findings that stakeholders can understand quickly.

Requirements - Ability to test AI systems whose outputs vary, using evaluation sets that reflect realistic user behavior. - Skill in designing regression tests across connected workflows and identifying silent downstream failures. - Judgment to assess edge cases and determine whether outputs are fit for business use. - Ability to measure and verify claimed outcomes with a repeatable, project-level method. - Attention to operational signals such as declining quality and increasing model costs. - Clear written communication for documenting findings, release decisions, and non-conformities.