Staff Applied Scientist

Role overview Lead the design and delivery of a rigorous evaluation framework for a production planner-executor agent. You will combine applied research with hands-on engineering, helping a customer determine which models and optimization strategies meet quality goals while improving cost and latency. The work spans experiment design, simulation, analysis, and a dashboard the customer can use to guide rollout decisions.

Responsibilities - Build an end-to-end evaluation workflow, including scenario formats, a multi-turn simulated user, isolated experiment runs, grading, and reporting. - Work with customer engineers to define scenario and grading standards, then validate the framework against production interaction logs. - Implement reusable optimization options, such as model cascades or caching, and instrument model calls to capture token use, latency, cache behavior, and cost. - Compare configurations on matched scenarios using confidence intervals and non-inferiority checks; clearly identify inconclusive results. - Translate measured quality and cost differences into production-scale projections and present the cost-quality trade-offs. - Deliver a locally deployable dashboard, run staged experiment campaigns, document integration guidance, and advise stakeholders on go/no-go decisions.

Requirements - Practical experience evaluating LLMs and multi-step, tool-using agents in production, including state-based verification, trajectory scoring, judging, or user simulation. - Strong experimental design and statistical inference skills, including paired comparisons, non-inferiority testing, and power analysis. - Experience with cost-aware ML or multi-objective optimization, including Pareto analysis and estimating business impact from deployment costs. - Understanding of model routing, inference behavior, and cost controls such as caching, context compaction, and reasoning budgets. - Experience with LLM tracing and observability, plus the ability to build reliable research tooling, data pipelines, and sandboxed execution in Python. - Strong customer-facing communication skills: you can scope research work, explain results to technical and business audiences, and lead engineering standards or mentor peers.

Nice to have - Publications or open-source work in agent evaluation, LLM systems, or efficient inference; experience with client-adoptable dashboards or LLM tracing tools; or work in regulated, high-stakes domains.