Senior Software Engineer – Open Source & SWE-Bench Evaluation

Role overview This remote, part-time consulting engagement centers on reviewing software engineering tasks built from real GitHub issues and pull requests in open-source projects. The work involves judging whether each task is technically sound, reproducible, properly tested, and representative of challenges professional engineers handle daily.

Responsibilities - Review coding tasks drawn from real bug fixes and feature implementations in open-source repositories - Assess whether problem statements and success criteria are clearly specified and complete - Evaluate unit tests for correctness, coverage, and robustness without unnecessarily constraining implementations - Identify flaky tests, missing dependencies, version conflicts, and environment-specific behavior - Determine whether tasks can be reliably reproduced across different setups - Provide clear recommendations on whether each task should be accepted, improved, or excluded

Requirements - Three or more years of professional software engineering experience - Strong background working with large, multi-file codebases - Experience reviewing pull requests, debugging, and maintaining production code - Solid understanding of unit testing, test coverage, dependency management, and reproducibility - Proficiency with Git and GitHub-based development workflows - Ability to analyze complex technical problems and articulate clear written feedback

Nice to have - Contributions to or maintenance of open-source projects - Familiarity with SWE-Bench, SWE-Bench Verified, or comparable coding benchmarks - Experience with major Python projects such as Django, Flask, scikit-learn, SymPy, matplotlib, requests, or pytest - Exposure to Docker, CI/CD, pip, conda, or dependency pinning - Knowledge of test fixtures, test isolation, or property-based testing - Background designing technical assessments or working on AI/ML evaluation, RLHF, or benchmark development

Benefits and work setup Remote part-time, project-based consulting engagement focused on open-source code review, testing, and technical evaluation.