Remote job
Member of Technical Staff | Knowledge Platform
Job details
About this role
Role overview Own the data systems that collect, process, and prepare information for research, model training, and production predictions. The work spans acquisition from websites and portals—including sources without APIs—and the streaming, batch, and lakehouse infrastructure that turns raw captures into reliable, versioned relational datasets.
Responsibilities - Build and operate crawlers, scrapers, and browser automation, with scheduling, retries, session handling, and change detection. - Monitor source coverage and freshness, identify changes in site layouts or access, and restore collection when it breaks. - Run event-driven ingestion using Kafka and Pub/Sub, handling ordering, replay, dead-letter queues, backpressure, and idempotent processing. - Develop batch and incremental data pipelines with Spark, dbt, and SQL to produce clean, typed relational tables. - Own lakehouse storage built around Parquet and support dependable datasets for downstream research and production use. - Operate the systems you build, including responding to incidents affecting sources or pipelines.
Requirements - Practical experience building collection systems for difficult sources and production data pipelines at scale. - Ability to work with event-driven ingestion and batch or incremental data processing. - Willingness to take operational responsibility and respond when systems fail. - Daily use of coding agents, paired with sound judgment about safe use.
Nice to have - Experience with JavaScript-heavy sites, authenticated sessions, rate limits, proxy management, or tools such as Playwright, Puppeteer, and Scrapy. - Workflow orchestration using Airflow, Dagster, or Temporal. - Entity resolution and relational or graph data modeling across multiple sources. - ML dataset practices such as point-in-time correctness, reproducible training sets, and evaluation splits. - Familiarity with cloud data services on GCP or AWS, OpenTelemetry, or financial and regulated environments.