Senior Data Scientist

Role overview Lead the evaluation and analysis of machine learning systems, including agents used in client work. You will turn model behavior and production data into evidence that guides engineering priorities and helps clients understand system performance, with substantial ownership of the evaluation approach.

Responsibilities - Define evaluation methods, datasets, metrics, rubrics, and release criteria for agents. - Build and calibrate automated evaluators, checking their results against human review. - Analyze production traces to identify and quantify failure patterns, then help prioritize fixes. - Design offline and online experiments to compare models, prompts, and system architectures. - Develop conventional models such as classifiers or forecasting models when they suit the problem better. - Work with engineers on monitoring dashboards and alerts, and explain results and trade-offs to non-technical stakeholders.

Requirements - At least three years of applied data science or machine learning experience with models in production. - Strong Python and SQL skills, with a sound understanding of statistics and experiment design. - Experience with NLP or language model systems and ways to evaluate them. - Ability to turn an unclear business question into a measurable analysis. - Clear writing and confidence in explaining what the data supports and where its limits are. - Ability to work independently and asynchronously in a small team.

Nice to have - Experience fine-tuning or adapting language models to specific domains. - Familiarity with retrieval, ranking, or search quality evaluation. - Experience with MLOps, monitoring, or data pipelines. - Consulting, agency, or other client-facing experience.

Benefits and work setup - Full-time, remote role with senior-level scope and direct collaboration with company leadership.