Copy of Senior Software Engineer, Data Acquisition

Role overview

Build and operate backend systems that collect, process, and deliver large scale datasets gathered from the web. You’ll improve the reliability, speed, observability, and data quality of ingestion systems while partnering with product and engineering teams to support existing and new data products.

Responsibilities

- Improve the architecture and operation of data acquisition and processing systems. - Develop and use web crawling tools to discover, capture, and catalog online information. - Build distributed services for job orchestration, data pipelines, and asynchronous workloads. - Structure collected information and monitor datasets for consistency, completeness, and quality. - Improve ingestion throughput, scalability, and fault tolerance, and work with partner teams on data product needs. - Write design documents, break complex work into milestones, and communicate progress and risks in a remote, asynchronous setting.

Requirements

- 7 or more years building or operating backend or infrastructure systems at scale. - Strong programming ability in Python, Go, Rust, or a similar language, including asynchronous or concurrent programming. - Understanding of distributed systems, including parallelism, backpressure, and message driven design. - Knowledge of browser rendering and web application fundamentals, including authentication, cookies, and HTTP. - Network debugging skills covering HTTP, DNS, proxies, and packet analysis; comfort with Linux or Unix command line and system resources. - Experience designing or maintaining resilient ingestion, API integration, or ETL systems, and evaluating data quality.

Nice to have

- Familiarity with queues and orchestration tools such as Kafka, SQS, or Airflow. - Experience with large scale crawling, indexing, or data ingestion; distributed processing or streaming platforms; SQL and data warehouses; cloud platforms; or modern storage patterns. - A degree in computer science, mathematics, engineering, or another quantitative field, or experience in security testing.

Benefits and work setup

- Remote first, with flexibility to work from anywhere. - Compensation listed at $160,000–$200,000, plus stock. - Benefits include unlimited paid time off, medical, dental, and vision coverage, and health, fitness, and office stipends.