About the company
Lead the engineering foundation behind reliable, measurable, and scalable AI-powered features. Build evaluation frameworks, observability tooling, and diagnostic infrastructure that reveal how AI agents perform in real production environments. This role combines hands-on software engineering with technical leadership and people management in a highly remote and asynchronous environment.
Responsibilities
- Design, build, and own evaluation infrastructure, including CI/CD pipelines, scorers, datasets, and systems for assessing AI agents
- Develop observability and diagnostic capabilities to identify quality issues across planning, execution, tool selection, and complex agent trajectories
- Investigate failures across AI workflows and turn findings into technical prototypes and priorities
- Build and expand datasets through human annotation, AI-generated examples, and simulated conversations
- Develop structured experimentation frameworks for prompts, models, and agent harnesses
- Identify opportunities to improve AI system cost and latency through model selection, caching, and routing
- Set technical direction and manage day-to-day priorities for the AI Quality engineering team
- Partner with AI Core engineering teams to ensure changes can be measured effectively
Requirements
- 7+ years of experience building and shipping production software, ideally including LLM-powered agents
- Experience with complex, tool-using AI systems involving multiple tools, planning, orchestration, or sub-agents
- Experience with Ruby on Rails and/or Python
- Experience building evaluation or observability infrastructure for ML/AI systems
- Familiarity with evaluation frameworks such as Braintrust, LangSmith, or similar tools
- Experience designing datasets, annotation workflows, or labeling pipelines
- Strong technical leadership and people-management capabilities
- Excellent English proficiency (CEFR C2 / ILR 5)
Conditions
- Annual cash compensation of $170,000 USD, benchmarked to U.S. compensation levels regardless of location
- Equity in the company, including ongoing refresh grants
- 35 days of paid time off per year
- Fully remote work environment
- Significant flexibility and autonomy
- Twice-yearly company retreats in international destinations
- Benefits supporting health, wellbeing, and professional development