QA Engineer, Voice AI Agents

Role overview This role centers on quality assurance for voice AI agents that engage patients in healthcare outreach and care-orchestration workflows. The engineer investigates live campaign failures, runs structured QA on agent conversations, and progressively builds automated checks to keep an agentic platform reliable at production scale. It blends hands-on debugging, campaign-level QA discipline, and growing tooling work in close partnership with engineering, deployment, and clinical operations teams.

Responsibilities - Triage and root-cause issues on live voice campaigns such as dropped calls, incorrect agent responses, failed handoffs to live staff, scheduling mistakes, and data mismatches - Trace problems through call recordings, system logs, campaign setup, and transcripts, then file clear, reproducible bug reports with severity and member impact - Run pre-launch QA on new campaigns and workflow changes, covering scripts, edge cases, escalation paths, and live-staff handoffs - Sample live conversations against defined quality standards and flag negative trends before they escalate into wider incidents - Help build and maintain the QA framework, including test cases, scoring rubrics, regression suites, and release checklists - Automate repetitive checks (transcript analysis, configuration validation, outcome reconciliation) and report quality metrics across deployment, product, and engineering

Requirements - 3 to 5 years in QA, support engineering, technical operations, or a comparable role on a production software platform - A strong debugging instinct with the patience to look past the first plausible explanation - Comfort reading logs, writing SQL queries, and working with APIs and JSON or YAML configuration - Track record writing test plans and bug reports that engineers trust - Clear written communication and disciplined documentation habits - Sound judgment on severity and prioritization when multiple live issues compete for attention

Nice to have - Hands-on experience with voice AI, conversational AI, IVR, or contact center platforms - Scripting ability in Python or similar for automating checks and analysis - Exposure to LLM evaluation, prompt testing, or conversation quality scoring - Healthcare experience, especially payer or provider outreach, and familiarity with HIPAA and PHI handling - Familiarity with observability or call analytics tooling

Benefits and work setup - Annual base salary range of $180,000-$200,000 plus meaningful equity, with total compensation benchmarked against top-tier tech employers - Generous paid time off, comprehensive health, vision, and dental insurance, plus life and disability coverage - Retirement plan with employer matching and a professional development budget for conferences, courses, and certifications - Wellness programs supporting physical and mental health - Remote or onsite options in New York City or Boca Raton, within a fast-paced, collaborative environment that ships often