Model Test and Measurement Engineer

Role overview

Independently evaluate scoring and generative AI systems used to organize records and support human review. This role establishes the evidence behind performance and release decisions through labeled datasets, sound sampling, version-aware evaluations, and clear measurement of both model behavior and its effect on review workflows. It works alongside model-building teams while preserving independent judgment over evaluation findings.

Responsibilities

- Build labeled ground-truth datasets and design and conduct statistically sound sampled audits. - Track model versions, evaluation records, and rollback triggers so release assessments are tied to specific versions. - Review drift and user overrides, and document human oversight and fairness or disparate effects. - Measure workflow outcomes such as review time, throughput, backlog movement, overrides, and rework. - Set quality-control methods for evaluation, including review units, population-aware sampling, and fair comparisons across differing methods or source sets. - Define required measurement events and confirm the review workspace captures them from initial use. - Provide independent evidence without building the models, setting thresholds, or authorizing releases based on the same evidence.

Requirements

- At least five years with model validation as a defined responsibility. - Direct experience constructing ground truth and designing sampling methods. - Strong Python and SQL skills, and the ability to work independently from the teams whose models are being evaluated. - U.S. citizenship and an active Secret clearance. - Ability to meet additional personnel security requirements for some work; details are covered during screening.

Nice to have

- Model-risk experience in government or regulated industries. - Fairness or disparate-effect testing and human-oversight documentation. - Familiarity with the NIST AI Risk Management Framework or a comparable practice. - An active Top Secret clearance.

Benefits and work setup

The role is fully remote with occasional travel to offices, customer sites, and partner facilities. The listed compensation range is $150,000–$190,000, with potential bonus and equity. Benefits include employer-paid medical, dental, and vision coverage for employees and their families, manager-approved unlimited paid time off, flexible scheduling, and 14 weeks of paid parental leave.