Technical Project Manager

Role overview

Manage technically complex Applied AI programs involving multilingual data collection, human-in-the-loop operations, and large language model evaluation. The role translates AI requirements into executable workflows, coordinates engineering and contributor teams, and uses operational metrics to deliver reliable datasets and evaluation results at the required quality, speed, and cost.

Responsibilities

- Lead data-collection and evaluation initiatives from technical scoping through implementation, validation, delivery, and retrospective review. - Convert AI requirements into data specifications, annotation guidance, evaluation plans, acceptance criteria, schemas, metadata requirements, and integration needs. - Coordinate supervised fine-tuning, preference-data, model-evaluation, safety-testing, and response-ranking workflows with technical specialists and domain experts. - Partner with engineers on ingestion, annotation tooling, data-generation pipelines, automated checks, system integrations, and issue resolution. - Operationalize hybrid evaluation approaches that combine human review, automated metrics, and LLM-as-a-judge methods. - Monitor throughput, quality, completeness, inter-annotator agreement, benchmark discrepancies, delivery risks, and unit economics; lead root-cause analysis when performance falls short.

Requirements

- Experience managing complex technical projects involving data operations, AI, machine learning, language technology, or related workflows. - Strong understanding of LLM training and evaluation concepts, including supervised fine-tuning, reinforcement learning from human feedback, human evaluation, automated evaluation, LLM-as-a-judge, and red teaming. - Proficiency in SQL for analyzing delivery velocity, quality metrics, and cost structures. - Working knowledge of data pipelines, structured data formats, APIs, validation processes, and engineering dependencies. - Ability to manage complex workflows using Agile, Scrum, or Kanban practices and communicate technical requirements to global teams. - Experience optimizing delivery economics, such as cost per task or token, while maintaining quality standards.

Nice to have

- Python or scripting experience for data analysis, automation, or evaluation tooling. - Experience managing specialized subject-matter experts or distributed contributor networks. - Familiarity with annotation platforms, business-intelligence tools, automated evaluation systems, and Jira. - Experience delivering multilingual or multimodal data programs, plus fluency in an additional language. - Background in computer science, data science, engineering, or equivalent practical experience.