Staff Data Engineer (Azure Databricks Platform)

Role overview Lead the architecture, operation, and ongoing improvement of a cloud data platform built on Azure Databricks. This hands-on staff role combines platform engineering, security and governance, performance and cost management, automation, and developer support, while directing the work of two platform engineers.

Responsibilities - Maintain the health of the platform through monitoring, logging, alerting, incident response, and root cause analysis. - Set platform direction and improve core services, including data storage, governance, and workflow capabilities. - Enforce data security and governance controls, including fine-grained access, lineage, and regulatory compliance. - Tune workloads and clusters, and provide tools and guidance that improve performance and control cloud costs. - Automate provisioning and resource management with infrastructure-as-code practices and CI/CD pipelines. - Create reusable patterns, mentor data engineers, provide technical support, and coordinate with business, security, networking, and infrastructure teams.

Requirements - At least eight years in hands-on engineering focused on cloud data platforms, distributed systems, or data engineering. - Deep practical experience administering Azure Databricks, including clusters, job orchestration, Delta Lake, and Unity Catalog. - Expert knowledge of Azure services such as Data Lake Storage, identity, and networking. - Strong skills in Terraform, Python, Spark or PySpark, SQL, and shell scripting. - Experience building infrastructure-as-code solutions and automated CI/CD pipelines for platform or data deployments. - Strong problem solving, communication, documentation, and stakeholder influence skills.

Nice to have - Experience with Azure Synapse, Kafka, or Snowflake, and familiarity with healthcare data or insurance feeds. - A degree in computer science, engineering, or a related field; relevant Azure or Databricks certifications.

Benefits and work setup The full-time role offers a stated typical annual pay range of $130,295–$260,590, with eligibility for an incentive program and an equity award target. Benefits include medical, dental and vision coverage, paid time off, retirement savings options, wellness programs, and other resources, subject to eligibility.