About the company
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Software Engineer L5/L6 — Model Evaluations & Data Curation (MEDC) based in the United States.
Responsibilities
- Design and build reusable data curation infrastructure, including shared libraries, components, and workflows that replace fragmented notebook-based processes
- Develop scalable LLM-powered pipelines that transform raw catalog, metadata, and other data sources into training and evaluation datasets such as question-answer pairs and synthetic scenarios
- Implement large-scale batch inference workflows while balancing data quality, computational efficiency, token usage, and cost
- Develop sampling strategies that optimize coverage, diversity, difficulty, and representation across relevant content and member segments
- Create data-quality and filtering systems using techniques such as LLM-as-judge scoring, evaluation-model-based ranking, deduplication, validation, and other quality controls
- Partner closely with researchers to design experiments that measure how data curation choices affect downstream model behavior and performance
- Establish curated datasets as discoverable, reusable artifacts with clear versioning, lineage, documentation, and reproducibility
- Drive adoption of standardized data curation practices across engineering, research, and modeling teams
- At the L6 level, provide technical leadership across data and evaluation infrastructure and help define technical direction for multi-engineer initiatives
Requirements
- Strong software engineering expertise in Python, including experience developing reusable infrastructure, libraries, frameworks, or platforms used by other engineers and researchers
- Hands-on experience building LLM-driven data generation or transformation pipelines, including synthetic data generation, structured outputs, or large-scale batch inference
- Practical experience with data quality techniques such as sampling, filtering, deduplication, validation, and model-based quality scoring, including LLM-as-judge approaches
- Strong modeling intuition and an understanding of how dataset composition and curation decisions can influence model behavior and performance
- Experience designing experiments or evaluation approaches to measure the impact of data and modeling decisions
- Experience with distributed data processing technologies such as Spark, Ray, or comparable frameworks
- Excellent collaboration and communication skills, particularly when partnering with researchers, data scientists, and platform engineering teams
- For L6 roles, demonstrated experience with LLM evaluation systems is required
- For L6 roles, demonstrated technical leadership across data or evaluation infrastructure, including setting technical direction for multi-engineer initiatives, is required
- Experience with dataset versioning, lineage, artifact management, experiment tracking, or model registries is highly valued
- Experience optimizing large-scale LLM inference for cost, throughput, or operational efficiency is a plus
- Familiarity with human annotation workflows and methods for calibrating LLM judges against human ratings is beneficial
- Experience with pipeline orchestration frameworks such as Metaflow, Airflow, or similar tools is advantageous
- Background in recommendation systems, personalization, search, content catalogs, or metadata-driven applications is a plus
Conditions
- Annual compensation range of $600,000–$1,066,000, with the range varying based on location and individual market factors
- Compensation is structured primarily around annual salary, with the flexibility to determine the desired balance between salary and stock options each year
- Comprehensive health insurance plans and mental health support
- 401(k) retirement plan with employer matching
- Stock option program
- Health Savings Accounts and Flexible Spending Accounts
- Family-forming benefits
- Life and serious injury benefits
- Disability programs
- Paid leave of absence programs
- Flexible paid time off for full-time salaried employees
- Remote work opportunity within the United States
- Opportunity to work on high-impact AI infrastructure spanning foundation models, evaluation, and data curation
- Collaborative environment with substantial technical autonomy and opportunities for senior-level technical leadership