Member of Technical Staff ML Research Engineer Pre Training Data

You will research, source, curate, and prepare web-scale and public data for model pre-training. You will build and own production pipelines for filtering, deduplication, decontamination, and data lineage. You will design data mixtures and curricula, create synthetic data using LLMs and distillation, run proxy experiments and ablations, and build trustworthy evaluation sets to measure model-quality improvements.

Responsibilities

• Source web-scale and public datasets
• Build pipelines to filter, deduplicate, and decontaminate datasets
• Design data mixtures and curricula for pre-training and midtraining
• Decide which data goes into training runs
• Build synthetic data generation systems using LLMs, structured prompting, and distillation
• Run proxy experiments and ablations to measure model-quality effects
• Build trustworthy evaluation sets
• Maintain clear, reproducible data lineage for training runs

Requirements

• Hands-on experience with pre-training data work, including web-scale sourcing and curation, synthetic data generation, mixing and curriculum design, deduplication, and decontamination
• Ability to reason about the effects of data decisions on training, model size, architecture, and training dynamics
• Experience designing experiments to test data changes, including baselines, scaling ladders, small-scale proxies, and held-out loss
• Experience building and owning production data pipelines
• ML fundamentals and experience training, evaluating, and iterating on models in Python
• 3+ years of relevant experience with an M.S., 1+ year with a Ph.D., or 5+ years with a B.S.

Benefits

• Equity
• Unlimited PTO
• Company-wide Refill Days
• Comprehensive health coverage
• Paid leave for pregnancy, parenting, medical needs, and family care