Senior Software Engineer – Open Source & SWE-Bench Evaluation

Role overview Evaluate software engineering tasks drawn from real open-source issues and code changes. You’ll judge whether each task is clearly defined, technically realistic, reproducible, and supported by tests that verify a solution without unnecessarily limiting how it can be implemented. The work is remote, part-time, and project-based.

Responsibilities - Review tasks based on real bugs, feature requests, repositories, and pull requests. - Check problem statements, expected outcomes, and acceptance criteria for clarity and completeness. - Assess unit tests for correctness, coverage, reliability, and appropriate flexibility. - Identify flaky tests, missing dependencies, version conflicts, and environment-specific behavior. - Evaluate setup instructions and determine whether tasks can be reproduced across environments. - Rate task complexity and provide recommendations to accept, revise, or exclude it.

Requirements - At least 3 years of professional software engineering experience. - Confidence working in large, multi-file codebases and maintaining production software. - Experience debugging issues, reviewing code changes, and using Git-based workflows. - Strong knowledge of unit testing and test coverage, including how to assess whether tests validate the intended behavior. - Experience with dependency management, environment configuration, and reproducibility. - Ability to analyze complex technical issues and write clear, actionable feedback.

Nice to have Open-source contributions or maintenance experience; familiarity with coding benchmarks; experience with Python open-source projects, Docker, CI/CD, pip or conda, dependency pinning, test isolation, fixtures, or property-based testing. Experience with technical assessments or AI evaluation is also useful.