About the company
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a SWE-Bench AI Task Auditor - Freelance AI Trainer Project based in Norway.
Responsibilities
- Evaluate software engineering tasks for technical accuracy, realism, solvability, reproducibility, and alignment with SWE-Bench standards.
- Review task codebases, integrations, tests, and evaluation criteria to identify potential technical weaknesses.
- Rigorously test and troubleshoot complex technical scenarios to determine whether tasks function as intended.
- Investigate codebase integration problems, test failures, logic errors, and other implementation issues.
- Provide clear, precise, and actionable feedback that enables task creators to correct identified problems.
- Apply professional software engineering judgment to assess whether tasks reflect realistic development scenarios.
- Help maintain a high standard of technical rigor and accuracy across AI training and evaluation workflows.
Requirements
- Demonstrable professional experience in software engineering, including experience navigating complex codebases and developing real-world applications.
- Strong knowledge of software engineering principles and familiarity with SWE-Bench-style tasks and evaluation workflows.
- Strong analytical and problem-solving abilities, with the capacity to investigate complex technical scenarios systematically.
- Experience troubleshooting code, diagnosing test failures, and identifying underlying logic or integration issues.
- Ability to evaluate technical work objectively and communicate findings clearly and constructively.
- Strong attention to detail and a rigorous approach to technical validation.
- Ability to work independently and manage project-based assignments effectively.
- Deep expertise in one relevant software engineering specialty is sufficient; expertise across multiple domains is not required.
- A secure computer and reliable, high-speed internet connection suitable for remote technical work.
Conditions
- Fully remote freelance contract.
- Flexible project-based working environment.
- Opportunity to contribute directly to the training and evaluation of AI systems.
- Ability to apply real-world software engineering expertise to challenging technical tasks.
- Compensation of $60/hour, with the final rate determined based on experience, expertise, and geographic location.
- No company-sponsored health insurance, PTO, or other employee benefits, as this is a freelance contractor position.