About the company
Our partner is a company building production-grade AI agents that help enterprises monitor, troubleshoot, and improve critical AI systems.
Responsibilities
- Lead AI initiatives end-to-end: Take ambiguous technical and product problems from initial research and experimentation through prototyping, validation, and production deployment, quickly identifying approaches that do not work.
- Build autonomous AI agents: Design and ship agent-powered capabilities such as root-cause analysis, incident triage, and automated monitoring, integrating them into production systems alongside engineering teams.
- Develop AI evaluation infrastructure: Create and maintain golden datasets, regression suites, offline and online scoring systems, and evaluation frameworks that enable reliable iteration on non-deterministic AI systems.
- Own retrieval and context pipelines: Build pipelines leveraging customer metadata, lineage, query history, and other relevant context while ensuring agent behavior is observable and measurable in production.
- Monitor and improve agent performance: Establish tracing, failure taxonomies, cost and latency budgets, and other operational signals to continuously improve quality, reliability, and efficiency.
- Collaborate cross-functionally: Partner with data science teams on detection quality and experiment design, and with product managers to determine what AI agents should do based on customer and business needs.
- Establish engineering standards: Define reusable patterns, guardrails, internal tooling, and best practices for building safely and effectively with LLMs and agentic systems.
- Promote AI-first engineering: Use modern AI development tools as part of everyday coding, research, experimentation, and engineering workflows.
Requirements
- Production agent experience: Demonstrated experience independently building and operating autonomous AI agents in production, including systems where models use tools, make decisions, and determine subsequent actions without continuous human intervention.
- AI evaluation expertise: Proven experience designing and owning evaluation frameworks for non-deterministic AI systems, including golden datasets, regression testing, offline and online scoring, and post-launch monitoring.
- Strong Python and ML foundations: Advanced proficiency in Python, with a solid backend engineering foundation and meaningful experience in machine learning, data science, or a closely related discipline.
- Model understanding: Ability to reason about how AI and language models behave beyond simply integrating third-party APIs or frameworks.
- Problem-solving mindset: Comfortable working from ambiguous problem statements rather than detailed specifications, with the ability to design experiments, build focused prototypes, assess results, and move successful solutions into production.
- Modern AI fluency: Regular user of AI development tools such as Claude or comparable solutions for coding, research, experimentation, and engineering productivity.
- Bias toward execution: Able to prioritize shipping practical, reliable solutions over unnecessary polish while recognizing when a problem requires deeper technical investigation.
- Backend focus: Comfortable working primarily at the backend and model layers rather than frontend development.
- Communication and collaboration: Able to work effectively with engineering, data science, and product stakeholders and communicate technical trade-offs clearly.
- Nice to have: Experience with applied statistics and hypothesis testing, building or maintaining MCP servers, or working with modern data and cloud technologies such as Snowflake, Databricks, dbt, or Airflow.
Conditions
- Competitive annual base salary: $180,000–$240,000, with final compensation determined by skills, experience, and location.
- Equity: Eligibility for stock options as part of the overall compensation package.
- Healthcare: Comprehensive healthcare plans.
- Retirement: 401(k) retirement plan.
- Flexible time off: Flexible paid time off designed to support sustainable working practices.
- Parental support: Paid parental leave.
- Remote-first flexibility: Work remotely across the Americas with a flexible, distributed work environment.
- Home office support: Home office stipend to help create an effective workspace.
- Connectivity reimbursement: Cell phone or Wi-Fi reimbursement.
- Wellness support: Wellness stipend to support physical and mental well-being.
- Travel: Generous travel policy.
- Inclusive culture: A diverse, collaborative environment that values different backgrounds, perspectives, beliefs, and experiences.
- High-impact work: Opportunity to build emerging AI technologies in a rapidly developing market and deliver solutions used by enterprise customers.
- Professional growth: Work alongside experienced engineering, data science, and product professionals while contributing to new technical standards and practices.