Remote job
Staff Data Engineer | Allconnect
Job details
About this role
Role overview Lead the design and modernization of large-scale data platforms, with a focus on Databricks and the infrastructure that supports machine learning and generative AI. You’ll work across engineering teams and external technical partners to provide reliable, governed and cost-conscious data inputs for production models.
Responsibilities - Turn stakeholder needs into platform architecture decisions and guide technical design across teams. - Modernize and operate Databricks environments used by multiple engineering groups. - Build AI and machine learning data infrastructure, including feature stores and vector databases. - Develop pipelines for embeddings, retrieval-augmented generation, and versioned training data. - Set standards for governance, data quality, observability, access control and protection of personal information. - Improve platform cost efficiency and delivery practices through CI/CD, infrastructure automation and data contracts. - Mentor engineers and lead design reviews to strengthen technical quality.
Requirements - 10+ years of data engineering experience designing and operating large-scale platforms. - Production experience with Databricks, SQL, Spark and Python. - Experience using Terraform to manage data platform infrastructure. - Familiarity with workflow orchestration tools such as Airflow or Dagster. - Experience with streaming or messaging platforms, such as Kafka, Kinesis or similar. - Knowledge of data governance, quality and observability practices, including access controls and PII safeguards. - Experience with DevOps and FinOps practices, such as CI/CD, data contracts and platform cost optimization. - Ability to explain technical trade-offs to non-engineering stakeholders and guide discussions with external partners.
Benefits and work setup The role is not available for visa sponsorship or transfer, or on a corporation-to-corporation basis. Benefits, subject to eligibility, include medical, dental and vision coverage, life and disability insurance, flexible spending accounts, a matched 401(k), employee assistance, paid parental bonding leave and flexible paid time off. The listed annual cash compensation is $150,000–$220,000, with a separate New York City range of $162,000–$275,000; actual compensation depends on location and qualifications.