Applied Research Engineer

Role overview Develop research-informed methods that make it easier to evaluate applications built with large language models. The work spans optimization, synthetic test data, and adapting models to serve as evaluators, with a focus on making automated assessments align more closely with human judgments.

Responsibilities - Design optimization algorithms to improve agreement between model-based evaluators and human assessments. - Create methods for generating synthetic test cases for language model applications, including retrieval-augmented and tool-using workflows. - Fine-tune and align language models for use as custom evaluators. - Apply findings from research to practical evaluation problems and develop new solutions.

Requirements - Extensive experience working with large language models. - Experience with synthetic data generation and optimization. - Interest in reading and critically assessing research while remaining open to new approaches. - Evidence of interest in open source or competitive data science, such as contributions to open-source projects or Kaggle participation.

Benefits and work setup - Remote role. - Advertised compensation range: $60,000–$200,000, plus 0.5%–2.5% equity.