SWE-Bench AI Task Auditor - Freelance AI Trainer Project

Accountabilities

• Evaluate software engineering tasks for technical accuracy, realism, solvability, reproducibility, and alignment with SWE-Bench standards.
• Review task codebases, integrations, tests, and evaluation criteria to identify potential technical weaknesses.
• Rigorously test and troubleshoot complex technical scenarios to determine whether tasks function as intended.
• Investigate codebase integration problems, test failures, logic errors, and other implementation issues.
• Provide clear, precise, and actionable feedback that enables task creators to correct identified problems.
• Apply professional software engineering judgment to assess whether tasks reflect realistic development scenarios.
• Help maintain a high standard of technical rigor and accuracy across AI training and evaluation workflows.

Requirements

• Demonstrable professional experience in software engineering, including experience navigating complex codebases and developing real-world applications.
• Strong knowledge of software engineering principles and familiarity with SWE-Bench-style tasks and evaluation workflows.
• Strong analytical and problem-solving abilities, with the capacity to investigate complex technical scenarios systematically.
• Experience troubleshooting code, diagnosing test failures, and identifying underlying logic or integration issues.
• Ability to evaluate technical work objectively and communicate findings clearly and constructively.
• Strong attention to detail and a rigorous approach to technical validation.
• Ability to work independently and manage project-based assignments effectively.
• Deep expertise in one relevant software engineering specialty is sufficient; expertise across multiple domains is not required.
• A secure computer and reliable, high-speed internet connection suitable for remote technical work.

Benefits

• Fully remote freelance contract.
• Flexible project-based working environment.
• Opportunity to contribute directly to the training and evaluation of AI systems.
• Ability to apply real-world software engineering expertise to challenging technical tasks.
• Compensation of $60/hour, with the final rate determined based on experience, expertise, and geographic location.
• No company-sponsored health insurance, PTO, or other employee benefits, as this is a freelance contractor position.