Remote job
Clinical AI Quality Specialist
Job details
About this role
Role overview
Assess the clinical quality and safety of an AI assistant designed to support people managing cardiovascular health. You will evaluate behavior before and after releases, review real user conversations, and work with product and engineering partners to turn clinical findings into clear, trackable decisions.
Responsibilities - Create test cases, including ambiguous and challenging scenarios, and score system responses against expected clinical behavior. - Build and maintain evaluation datasets and contribute to methods for scoring results. - Review sampled production conversations on a weekly basis and help prepare monthly reports that explain the scope and limits of the review. - Assess handling of emergencies, self-harm risk, and medication safety, including whether escalation is excessive or insufficient. - Document failures, likely contributing factors, and clinical risk in terms engineering can act on; ensure findings have an owner, ticket, and decision. - Maintain current, auditable clinical specifications, review records, and supporting evidence.
Requirements - PharmD or equivalent clinical doctorate, an active U.S. license, and at least three years of combined clinical practice and clinical AI evaluation. - Experience evaluating large language model systems, reviewing production conversations, and working with monitoring tools or similar workflows. - Strong knowledge of triage, escalation, crisis and self-harm response, and medication safety based on current guidance. - Ability to investigate assumptions, consult and cite guidelines, drug labels, and primary sources, and verify AI-generated material before relying on it. - Clear written and verbal communication, including the ability to explain clinical risk to product and engineering teams and maintain precise documentation. - Comfort with ambiguity, feedback, and evolving AI evaluation practices.
Nice to have
Experience in cardiovascular care or hypertension management, structured clinical transcript rating and rater agreement, crisis or self-harm review, or use of monitoring tools such as Langfuse.
Benefits and work setup
The stated U.S. base salary range is $130,000–$143,000 annually. The posting says this figure covers salary and bonus, and excludes equity and benefits.