About the company
Newcode.ai is a fast-growing legal tech and agentic AI company transforming how legal work is done. With teams across Norway, Sweden, Ireland, the US and growing we work at the intersection of law, technology, and intelligence. We move fast, think big, and take pride in doing things the right way.
Responsibilities
- Design and build automated and human-in-the-loop evaluation frameworks for AI-powered products and workflows.
- Develop metrics and benchmarks to measure accuracy, reliability, consistency, and overall AI quality.
- Build evaluation datasets, test cases, and regression suites for complex legal use cases.
- Analyze model outputs to identify failure modes, patterns, and opportunities for improvement.
- Work closely with Legal Engineers to translate real-world legal workflows into rigorous evaluation criteria.
- Partner with Software Engineers to integrate evaluations directly into product and development workflows.
- Establish automated testing and monitoring to identify quality regressions before they reach customers.
- Evaluate changes to models, prompts, retrieval systems, agents, and workflows.
- Develop tooling that makes AI quality measurable and visible across the organization.
- Investigate unexpected model behavior and determine root causes.
- Help establish standards for what "good" looks like across different legal use cases.
Requirements
- Strong software engineering or technical background, with experience working on AI/ML systems.
- Experience building testing, evaluation, or quality frameworks for AI-powered products.
- Strong Python and/or similar programming experience.
- Understanding of LLMs, generative AI, retrieval systems, agents, and/or AI evaluation methodologies.
- Strong analytical and problem-solving skills.
- Ability to work with ambiguous problems and turn them into measurable technical frameworks.
- Strong attention to detail and a high bar for quality.
- Ability to communicate technical findings clearly to both technical and non-technical stakeholders.
- Comfort working cross-functionally with Engineering, Product, and Legal Engineering.
Conditions
- Culture of trust and excellence: Be part of a team that values reliability, respect, and initiative, where everyone’s contribution matters and diversity drives creativity.
- Collaborative environment: Work with talented, driven peers on bleeding-edge legal engineering solutions with real-world impact.
- Growth and ownership: Shape your role as we grow. Take ownership and drive outcomes from the start of your journey.
- Inclusive culture of excellence and trust: We reward initiative, respect, integrity, and teamwork, and are committed to building a global, inclusive team. Your drive, curiosity, and ideas matter more than your background.