Senior Python Engineer

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Python Engineer - AI Code Evaluation based in the United States.

• Evaluate AI coding interactions: Review AI-generated coding sessions end to end and assess whether responses are useful, correct at a high level, and aligned with how an experienced engineer would approach the problem.
• Assess engineering judgment: Evaluate whether AI agents demonstrate sound reasoning, appropriate trade-offs, clear decision-making, and practical software-engineering judgment rather than simply producing syntactically correct code.
• Review explanations and reasoning: Judge the quality of model preambles, explanations, reasoning, and guidance, identifying whether they genuinely help developers or simply generate technically accurate but unhelpful output.
• Provide structured feedback: Clearly and directly explain what worked, what did not, and what felt misleading, ineffective, or inconsistent with strong engineering practice.
• Differentiate response quality: Apply consistent standards to distinguish varying levels of AI performance and articulate what separates an average response from an excellent one.
• Help define evaluation standards: Contribute to the evolving understanding of what high-quality AI-assisted software development should look like, including interactions with AI-first development environments.
• Apply independent judgment: Make subjective but rigorous assessments based on extensive engineering experience, without needing to execute or deeply inspect every line of generated code.

• Senior engineering experience: Staff-, Principal-, or similarly experienced software engineering background, with a strong track record of making technical decisions and maintaining a high standard for software quality.
• Strong programming expertise: Deep hands-on experience with Python and/or TypeScript/JavaScript, with the ability to quickly understand and assess modern software implementations.
• AI coding experience: Practical experience using AI-assisted development tools such as OpenAI Codex, Claude Code, Cursor, or comparable coding agents.
• Modern development knowledge: Strong familiarity with contemporary AI-assisted software-development workflows and how developers interact with coding agents.
• Engineering judgment: Ability to recognize whether an AI response reflects strong engineering thinking, including useful explanations, sound trade-offs, appropriate guidance, and trustworthy recommendations.
• Critical evaluation skills: Comfortable evaluating code and technical reasoning at a high level without needing to execute or inspect every implementation detail.
• Communication skills: Able to provide concise, direct, and opinionated feedback that clearly explains why an interaction succeeds or falls short.
• High quality bar: Strong personal standards for what constitutes effective, maintainable, and professional engineering.
• Preferred experience: Exposure to AI-first IDEs, prompt design, model evaluation, evaluation workflows, or engineering-standard development is a plus.
• Additional advantage: Experience mentoring senior engineers or helping establish engineering standards and best practices is beneficial.

• Compensation: $100–$200 per hour, depending on experience and fit.
• Flexible engagement: Approximately 10–20 hours per week.
• Remote work: Fully remote contract engagement with flexibility across eligible locations.
• Short-term focused engagement: Initial engagement runs through early May, with potential for extension.
• Fast start: Opportunity to begin as soon as possible.
• Meaningful AI impact: Directly contribute to evaluating and improving how AI coding agents support professional software engineers.
• Senior-level scope: Apply your engineering expertise to challenging questions around AI reasoning, developer experience, and software-engineering quality.
• Streamlined selection: Hiring process consists of a take-home evaluation exercise followed by one behavioral interview.