Remote job
Senior Research Engineer, AI Evaluation
Job details
About this role
Role overview Lead the design and implementation of evaluation systems for language models used in clinical decision support applications. The role combines applied research and engineering to measure model reasoning, retrieval, and structured responses, and to make quality and safety checks reproducible across development and deployment.
Responsibilities - Build and maintain automated evaluation frameworks for model outputs. - Develop general and domain-specific benchmarks, synthetic test cases, and trusted reference datasets. - Measure and address edge cases, hallucinations, model drift, and safety regressions. - Establish offline and online evaluation practices with research, infrastructure, and product partners, including integration into CI/CD workflows.
Requirements - At least three years in software engineering, applied machine learning, or AI research, with strong statistics and scientific rigor. - Demonstrated experience building LLM evaluation harnesses or benchmarks, such as model-based judging, reference metrics, or human review. - Advanced Python skills and experience with PyTorch or Hugging Face, model APIs, retrieval-augmented generation, and experiment tracking or observability tools. - Strong technical writing and communication skills.
Nice to have - Advanced degree in computer science, machine learning, statistics, or another scientific field. - Peer-reviewed research or presentations at machine learning conferences. - Knowledge graph experience.
Benefits and work setup - Base salary range of $180,000–$240,000, equity, health and wellness coverage, and paid time off. - Flexible work arrangement with a Lower Manhattan hub and support for remote work.