Position listed on behalf of a partner company that manages all applications and next steps.
Partner is looking for an Evaluations Team Lead based in Germany.
Responsibilities
Build and maintain a shared evaluation platform supporting research, engineering, and Applied AI teams.
Develop and continuously improve benchmark suites, curated datasets, and model comparisons.
Establish rigorous evaluation standards covering statistical metrics, validation strategies, data splits, leakage prevention, calibration, uncertainty estimation, and benchmark contamination.
Design reproducible evaluation pipelines with version-controlled inputs, datasets, and artifacts, integrating automated regression checks.
Benchmark models against competing approaches using fair and consistent protocols.
Evaluate predictive performance, inference latency, and cost efficiency across deployment configurations.
Support customer proof-of-concept evaluations with reliable tooling and methodological guidance.
Translate evaluation findings into actionable research priorities and release recommendations.
Recruit, lead, and develop a small team of evaluation specialists.
Remain hands-on with Python development, experimental design, statistical analysis, and evaluation methodology.
Requirements
Proven experience owning the evaluation of tabular machine learning systems used in production environments.
Strong statistical expertise, including selecting metrics and validation schemes, estimating uncertainty, and comparing models across datasets.
Practical experience identifying and preventing data leakage involving preprocessing, feature engineering, joins, temporal dependencies, and related entities.
Strong proficiency in Python and SQL, with familiarity with scikit-learn, gradient-boosted decision trees, and reliable ML tools or platforms.
Demonstrated experience designing fair and reproducible model comparisons, including hyperparameter optimization, resource allocation, and end-to-end inference latency measurement.
Previous people management experience, including hiring, technical mentoring, team development, and performance management.
Excellent written and verbal communication skills.
Strong analytical mindset and confidence to challenge unsupported conclusions.
Experience evaluating tabular foundation models or automated machine learning (AutoML) systems is an advantage.
Familiarity with measuring the impact of model optimizations on predictive performance and inference efficiency is desirable.
Experience with relational or multi-table datasets and data platforms such as Snowflake or Databricks is a plus.
Knowledge of automated or agent-driven machine learning workflows is beneficial.
Conditions
Competitive compensation package combining salary and equity.
Comprehensive health coverage for employees and their dependents.
Paid parental leave for all new parents, including adoptive and surrogacy journeys.
Relocation assistance for employees moving to join the team at one of the company's office locations.
Remote working opportunities across eligible locations in Europe and Israel.
Opportunity to establish evaluation standards and build a new function from the ground up.
Mission-driven, collaborative, and low-ego culture.