QA and AI Evaluation Engineer

Role overview Help improve the reliability of both a web-based agent-building platform and the customer-facing AI agents created with it. You’ll explore how features behave across browsers and devices, probe conversational agents with realistic and challenging scenarios, and turn findings into clear reports and automated checks that help teams catch problems before release.

Responsibilities - Test new builder features across browsers and devices before release. - Evaluate agents using realistic and adversarial conversations, documenting failure patterns. - Write reproducible bug reports with steps, expected and actual behavior, and supporting evidence. - Build automated end-to-end tests for high-priority user journeys. - Help establish practical quality criteria for deciding when an agent is ready to publish.

Requirements - At least two years of quality assurance experience on a web product. - Experience with test automation tools such as Playwright or Cypress. - Ability to produce clear reports that include reproduction steps and evidence. - Curiosity about AI failure modes and patience to investigate them.

Nice to have - Experience testing chatbots or large language model features. - Working knowledge of JavaScript or TypeScript.

Benefits and work setup Remote contract with flexible hours. Compensation starts at $3,300 per month.

The application exercise asks candidates to build and publish an agent, test both it and the builder, and submit a short test plan plus 10–15 bug reports with reproduction details and screenshots or recordings. Candidates should identify the three issues they would prioritize. No time limit is stated.