Senior QA Automation Engineer

Role overview

Own quality strategy for an AI-powered voice product that combines language models, speech services, and telephony. This is a manual-testing-heavy leadership role: you will establish coverage standards, design challenging end-to-end scenarios, guide automation priorities, and investigate failures across the conversation and communications stack rather than serving as the primary framework developer.

Responsibilities

- Participate fully in Scrum ceremonies and maintain clear testing artifacts and traceability. - Define end-to-end test strategy for multi-turn conversations, interruptions, ambiguous requests, escalation, recovery, and out-of-scope behavior. - Build evaluation approaches for variable model output using representative datasets, semantic or rubric-based assertions, and regression detection. - Test for hallucinations, weak grounding, prompt injection, guardrail bypasses, and unauthorized exposure of customer information. - Provision SIP endpoints and call scenarios, then isolate failures across language models, speech recognition, speech synthesis, media transport, and routing. - Expand automated coverage, connect suites to CI/CD release gates, and report metrics such as task completion, intent accuracy, latency, transcription quality, and escalation correctness.

Requirements

- At least five years of QA experience owning quality for a complex, integrated production product. - Practical SIP/VoIP testing experience, including signaling, call flows, media handling, softphones, or trunking; able to use traces or packet captures. - Strong Python skills sufficient to build and maintain automation with tools such as pytest, plus experience reviewing and debugging test suites. - Experience testing probabilistic or non-deterministic systems and defining meaningful assertions for variable results. - Strong analytical and scenario-design skills, with the ability to use quality metrics to support release decisions. - Hands-on experience with formal test management systems and Jira, plus professional spoken and written English with overlap with US Eastern hours.

Nice to have

- Experience with conversational AI, LLM evaluation, speech quality, retrieval grounding, WebRTC or media-layer testing, IVR/contact-center platforms, multilingual testing, or prompt analysis.

Benefits and work setup

- Remote work, B2B engagement, long-term growth, annual performance review, hardware support, professional training and conference funding, English classes, and referral incentives.