Member of Technical Staff Backend Software Engineer Search Crawler

You will lead projects that develop large-scale web crawlers, ingestion pipelines, and data-processing systems. You will build and optimize backend and frontend crawler components, including storage, retrieval, and data-management dashboards. You will architect scalable distributed systems, improve performance and reliability, and iterate on new approaches.

Responsibilities

• Lead projects developing large-scale web crawlers, ingestion pipelines, and data-processing systems
• Build, maintain, and optimize backend and frontend crawler-service components
• Ensure reliable, high-quality ingestion and processing of web pages
• Architect and implement scalable distributed systems and fullstack features
• Improve system performance, usability, and reliability

Requirements

• 5 years of software development experience
• Data structures and algorithms
• Python
• C++
• Rust
• Go
• High-load distributed services
• Cloud infrastructure
• Kubernetes
• AWS