Senior Data Platform Engineer

You will scale and automate petabyte-scale data processing pipelines, design execution layers, and optimize compute and GPU use. You will make datasets reproducible and traceable, maintain recovery and observability capabilities, and work with researchers to integrate datasets into training pipelines.

Responsibilities

• Scale and automate petabyte-scale data processing pipelines
• Design execution layers for pipeline stages with differing computational needs
• Optimize compute resources and GPU access
• Build reliable, recoverable, restartable, and observable pipelines
• Version datasets and transformation outputs for reproducibility and traceability
• Maintain dataset datasheets in line with data-governance policies
• Partner with researchers to integrate datasets into training pipelines

Requirements

• Bachelor’s degree in a relevant computing field
• 5 or more years designing, implementing, and managing large-scale distributed data processing systems or ML data frameworks
• Experience with Ray, Apache Spark, workflow orchestrators, Apache Arrow, or Parquet
• Experience owning data systems under throughput, reliability, and cost constraints
• Experience optimizing heterogeneous and GPU-accelerated workloads
• Experience with dataset versioning, lineage, and reproducibility tooling
• Experience with workload managers such as Ray, Kubernetes, or Slurm
• Familiarity with Docker or Enroot
• Familiarity with vector databases

Benefits

• Comprehensive health benefits including mental health and wellness management account
• 20 vacation days per year from the start date
• 4% employer retirement savings contribution with no employee match