Machine Learning Scientist ML Research

You will design and conduct experiments on AI model behavior, create metrics and evaluation protocols, and analyze large-scale human preference data. You will prototype research ideas, collaborate to scale findings into production, publish research, support model-provider testing, and strengthen the scientific integrity of evaluation tools and leaderboards.

Responsibilities

• Design and conduct experiments evaluating AI model behavior
• Develop evaluation metrics, methodologies, and protocols
• Analyze large-scale human voting and interaction data
• Implement and scale research findings with engineers
• Prototype and test research ideas
• Author internal reports and external publications
• Partner with model providers on evaluation questions and testing
• Contribute to the integrity and transparency of evaluation tools and leaderboards

Requirements

• PhD or equivalent research experience in machine learning, natural language processing, statistics, or a related field
• Experience training large-scale models, reward models, preference models, and fine-tuned LLMs
• Knowledge of RLHF, DPO, and contrastive learning
• Knowledge of machine learning, statistics, LLMs, Transformers, diffusion models, and deep learning
• Proficiency in Python and PyTorch, JAX, or TensorFlow
• Experience designing and analyzing statistically rigorous experiments
• Experience publishing ML, NLP, or AI evaluation research or contributing to open-source projects
• Experience with real-world usage data and evaluation metrics

Benefits

• Equity
• Medical benefits
• Dental benefits
• Vision benefits
• Additional health and wellness support programs