Principal Data Scientist

Role overview Partner with platform and product leaders to measure whether foundational technology and AI-powered user experiences deliver real outcomes. The work centers on defining useful metrics, evaluating conversational and agent-based systems, and making credible impact assessments when standard randomized experiments are unavailable. Findings will inform platform priorities and executive-level planning.

Responsibilities - Own measurement strategy from event instrumentation and data pipelines through dashboards and impact attribution. - Define progress, quality, and outcome metrics for a consumer-to-partner connection platform and other foundational systems. - Design repeatable evaluations for LLM-based and agentic products, including approaches such as LLM-as-a-judge. - Assess causal impact when clean A/B tests are not feasible, clearly stating assumptions and limits. - Distinguish product adoption and activity from improvements in quality, productivity, or business outcomes. - Partner with engineering, data, and product teams to make instrumentation reliable and metrics trusted.

Requirements - A PhD with 4+ years, master’s degree with 6+ years, or bachelor’s degree with 8+ years of experience in a quantitative field such as statistics, econometrics, data science, machine learning, or operations research. - Experience measuring complex production systems, such as conversational, LLM-based, or agent-driven products. - Knowledge of modern evaluation methods and judgment about which quality signals are reliable. - Experience assessing AI or agent tools against real outcomes, including productivity, quality, and cost. - Experience working with internal teams as customers in a platform-as-a-product model. - Fluency in causal and quasi-experimental methods, such as difference-in-differences, synthetic controls, interrupted time series, or switchback designs.