Machine Learning Lead (LLM)

Role overview Lead a small team building production LLM research and analysis tools, with a focus on measuring and improving the quality of AI outputs. You will set the team roadmap, develop senior data scientists, and remain hands-on in evaluation systems, judge models, and automated methods for improving prompts and retrieval.

Responsibilities - Lead and coach a team of senior data scientists through planning, prioritization, feedback, and professional development. - Set the roadmap with senior leadership, focusing on improvements that matter to product users. - Define quality standards and build evaluation systems using test sets, human labeling, and LLM-as-judge workflows. - Train and calibrate judge models against expert human assessments, monitoring reliability as products and underlying models change. - Use evaluation results to improve prompts, retrieval, examples, and agent workflows. - Guide choices among commercial models, fine-tuned open-weight models, and production deployment approaches. - Work with engineering and subject-matter partners to connect model development to user needs, and communicate tradeoffs to technical and nontechnical audiences.

Requirements - At least 6 years in machine learning, data science, or applied statistics, including at least 1 year as a manager or technical lead. - Experience shipping ML or LLM features to production. - Strong applied statistics foundation and proficiency in Python, SQL, and a modern ML framework such as PyTorch or JAX. - Comfort with cloud and production tooling, product judgment, and attention to end-user experience. - Clear, direct leadership and communication skills, including the ability to guide a team through changing priorities and ambiguous problems.

Benefits and work setup - Remote-first, with an office in New York City and regular team meetups; work is primarily aligned to East Coast hours. - Annual salary range: $165,000–$210,000, based on experience. - Benefits include medical, dental, and vision coverage, a 401(k) match, 20 weeks of paid parental leave, family-building support, and unlimited PTO. - Candidates must be authorized to work lawfully in the United States.