← Back to jobs

Remote job

Member of Technical Staff | Knowledge Platform

Data Engineer Full-time São Paulo, Brazil

Job details

Not specified Salary
São Paulo, Brazil Eligibility
Staff Experience
Full-time Employment

About this role

Role overview Own the data systems that collect, process, and prepare information for research, model training, and production predictions. The work spans acquisition from websites and portals—including sources without APIs—and the streaming, batch, and lakehouse infrastructure that turns raw captures into reliable, versioned relational datasets.

Responsibilities - Build and operate crawlers, scrapers, and browser automation, with scheduling, retries, session handling, and change detection. - Monitor source coverage and freshness, identify changes in site layouts or access, and restore collection when it breaks. - Run event-driven ingestion using Kafka and Pub/Sub, handling ordering, replay, dead-letter queues, backpressure, and idempotent processing. - Develop batch and incremental data pipelines with Spark, dbt, and SQL to produce clean, typed relational tables. - Own lakehouse storage built around Parquet and support dependable datasets for downstream research and production use. - Operate the systems you build, including responding to incidents affecting sources or pipelines.

Requirements - Practical experience building collection systems for difficult sources and production data pipelines at scale. - Ability to work with event-driven ingestion and batch or incremental data processing. - Willingness to take operational responsibility and respond when systems fail. - Daily use of coding agents, paired with sound judgment about safe use.

Nice to have - Experience with JavaScript-heavy sites, authenticated sessions, rate limits, proxy management, or tools such as Playwright, Puppeteer, and Scrapy. - Workflow orchestration using Airflow, Dagster, or Temporal. - Entity resolution and relational or graph data modeling across multiple sources. - ML dataset practices such as point-in-time correctness, reproducible training sets, and evaluation splits. - Familiarity with cloud data services on GCP or AWS, OpenTelemetry, or financial and regulated environments.

Skills detected in the listing

JavaScriptPythonSQLdbtApache AirflowData EngineeringAWSGCPKubernetesTerraform
Detected Nov 18, 2026
Last verified Nov 19, 2026

Hidden Jobs Access

Unlock application links

Read the full job details for free. An active Hidden Jobs Access subscription is required to open the original application link.

Weekly

$6.99 /week
  • Original application links
  • Full access to all MCP features
  • Cancel anytime

Monthly

$35.99 $17.99 /month
  • Original application links
  • Full access to all MCP features
  • 35% cheaper than weekly

Lifetime

$99.99 $49.99 /forever
  • Original application links
  • Full access to all MCP features
  • One-time payment
Hidden Jobs gives subscribers direct access to original application links