Senior AI Systems Quality Engineer

Role overview

Build the engineering practices and platform that make AI systems reliable and safe to operate in production. You’ll create automated ways to evaluate agentic and LLM-based workflows across their lifecycle, with an emphasis on repeatable quality controls, traceability, and graceful behavior when systems encounter uncertainty or failure.

Responsibilities - Build automated validation frameworks, test harnesses, and evaluation pipelines for AI systems. - Develop an AI testing platform integrated with Databricks and MLflow to support repeatable evaluation, lineage, and auditability. - Create large scenario-based test suites to cover end-to-end workflows, edge cases, and failure modes. - Evaluate orchestration, including tool use, memory, and decision logic, and stress-test non-deterministic behavior. - Define system contracts, guardrails, and safe-degradation patterns; establish measurable quality signals such as grounding, relevance, latency, and cost. - Integrate automated checks and quality gates into the development and deployment lifecycle.

Requirements - At least 7 years of software engineering experience, primarily in backend or platform systems. - Experience designing and implementing production AI testing automation and custom evaluation frameworks for complex systems. - Strong Python and/or TypeScript skills and hands-on experience with LLM-based or agentic systems. - Experience with large-scale regression testing, long-tail evaluation, or broad test coverage for AI systems. - Strong understanding of CI/CD and AWS cloud-native architectures. - Knowledge of reliability, governance, safety, and security or privacy risks in regulated or mission-critical environments.

Nice to have - Familiarity with testing non-deterministic AI outputs, drift detection, bias and fairness evaluation, and robust regression methods. - Experience with Databricks, MLflow, and AI evaluation platforms.