SeniorRemoteCanada
Data Scientist
О роли
Описание вакансии
About the company
- Partner company in the financial technology space, fast-growing environment.
Responsibilities
- Define ground truth, quality metrics, scoring methodologies, and evaluation frameworks for AI models and agents.
- Build repeatable evaluation loops to monitor model quality, identify regressions, and support release decisions.
- Apply statistical methods: sample sizing, confidence intervals, significance testing, handling non-deterministic outputs.
- Validate automated graders against human assessments.
- Partner with Engineering and Analytics Engineering to operationalize evaluation frameworks.
- Interpret evaluation results and provide actionable recommendations.
- Establish evaluation guidelines, documentation, and quality standards.
- Collaborate with Product, Engineering, Analytics Engineering, and business stakeholders.
- Promote evaluation best practices and mentor others.
- Support safe AI deployment through independent assessments.
Requirements
- 6–10 years in quantitative data science, machine learning, or related field.
- Strong statistical foundations and experiment design.
- Experience defining evaluation metrics and assessing complex model outputs.
- Proficiency in Python and SQL.
- Experience evaluating ML models in production.
- Ability to validate automated graders against human judgments.
- Strong problem-solving and analytical judgment.
- Excellent communication skills.
- Proven cross-functional collaboration and independent ownership.
- Quantitative degree is an advantage.
- Experience with LLMs or AI agents, evaluation harnesses, LLM-as-judge calibration, CI regression gates is a plus.
- Experience with text-to-SQL systems or analytics agents is an advantage.
- Background in fintech, brokerage, or financial services is desirable.
- Familiarity with AI tools for research and engineering workflows.
Conditions
- Competitive compensation.
- Stock options.
- Health benefits.
- Home-office setup allowance: one-time USD $500.
- Monthly stipend: USD $150 via Brex card.
- Remote work within the Americas, Canada target location.
- Technical ownership of evaluation practices.
- Cross-functional impact.
Стек и навыки
С чем работаем