Queue Manager / AI Data Quality Lead (Trust & Safety Background) - DP

Role overview This freelance, remote role sits at the intersection of AI data operations, quality assurance, and contributor management for a live training and evaluation program. The Queue Manager owns day-to-day execution for one or more annotation queues, balancing throughput with rigorous quality control and acting as a signal-quality analyst rather than a generic operations lead. A trust and safety or content integrity background is central, since the work depends on spotting shallow, low-effort, or misaligned responses before they corrupt training data.

Responsibilities - Operate the work queue end-to-end, handling intake, triage, prioritization, task assignment, backlog control, and meeting agreed service-level targets. - Run quality assurance on annotator output, correct submissions, deliver feedback, and flag systemic policy confusion or tooling issues to the client delivery lead. - Design and lead calibration sessions that drive guideline adherence, reduce rework, and stabilize quality at the project and individual contributor level. - Manage contributor performance at scale, including retention decisions for high performers and removal of underperformers, while supporting new joiners through onboarding. - Build and maintain dashboards that surface throughput, disagreement rates, defect rates, and contributor performance, and translate raw data into actionable reporting for stakeholders. - Train new contributors, run live or recorded orientation webinars, host office hours, and staff dedicated support channels during a project.

Requirements - Direct, hands-on experience running queues and quality on AI data programs, ideally with prior exposure to a major data labeling or evaluation provider. - Three or more years in high-throughput operations such as data labeling, trust and safety, business process outsourcing, content operations, or customer experience operations, with recent AI-specific work. - Demonstrated ability to design calibration processes from scratch and to identify shallow or low-effort responses as part of routine quality work. - Track record building dashboards and reporting using spreadsheets, with SQL, business intelligence, Looker, or Tableau experience viewed as a strong plus. - Comfort operating under ambiguity and shifting priorities on live programs, including coordinating distributed contributors across time zones and contractor or vendor workforces.

Nice to have - Familiarity with machine learning data or large language model evaluation workflows such as ranking, rubric-based judging, gold sets, and iterative calibration. - Prior consulting-style work advising clients on AI training methodology rather than only executing tasks.

Benefits and work setup - Fully remote freelance engagement with English as the working language and flexible scheduling once onboarded. - Short onboarding path after a qualifying assessment, with hourly compensation proposed by the candidate and competitive rates tied to AI data programs.