This position is listed on behalf of a partner company, who manages all applications and next steps.
Responsibilities
Evaluate developer workflow tasks for technical accuracy, realism, solvability, reproducibility, and completeness.
Assess whether tasks are supported by reliable tests and appropriate evaluation criteria.
Audit AI-assisted developer workflows and identify issues that could affect the quality or validity of training and evaluation tasks.
Review IDE integrations and identify technical faults, inconsistencies, or workflow issues.
Analyze AI-assisted coding processes and identify inefficiencies, logic errors, or technical shortcomings.
Test and troubleshoot complex technical scenarios to determine whether tasks function as intended.
Provide clear, actionable, and technically detailed feedback on identified issues.
Apply specialized knowledge of developer tooling, AI coding assistants, telemetry, trace analysis, and developer workflow ecosystems within the assigned area of expertise.
Help ensure AI training tasks meet rigorous technical standards and accurately reflect practical developer workflows.
Requirements
Demonstrable professional experience and deep technical knowledge in modern developer tooling.
Experience with AI-assisted coding tools, developer workflow ecosystems, telemetry, trace analysis, or closely related technical domains.
Strong analytical and problem-solving skills, with the ability to rigorously test and troubleshoot complex technical scenarios.
Ability to assess technical tasks objectively for accuracy, practicality, reproducibility, and reliability.
Strong understanding of software development workflows and the technical challenges involved in AI-assisted development.
Ability to identify IDE integration issues, coding inefficiencies, logic errors, and other technical faults.
Strong attention to detail and ability to provide precise, actionable technical feedback.
Candidates will be considered for a specific specialty based on their professional experience; expertise across all listed domains is not required.
Ability to work independently in a remote freelance environment and deliver high-quality technical assessments.
Conditions
Fully remote freelance contract.
Pay rate of $60 per hour, with the exact rate determined based on experience, expertise, and geographic location.
Opportunity to contribute technical expertise to the development and evaluation of advanced AI systems.
Work focused on modern AI-assisted developer workflows and emerging software development technologies.
Opportunity to influence the technical quality, reliability, and practical relevance of AI training tasks.
Flexible remote working environment.
Contractors provide their own secure computer and high-speed internet.
Company-sponsored benefits such as health insurance and paid time off are not included for contractors.