Databricks Architect

Role overview A national third-party benefits administrator is hiring a Databricks Architect & Developer to own the end-to-end design of an enterprise lakehouse on Azure. The work spans ingestion, transformation, governance, and developer enablement across multiple claims systems, partnering with engineers, analysts, and business stakeholders to deliver production-ready data platforms.

Responsibilities - Architect and document scalable lakehouse solutions using Databricks, Apache Spark, Delta Lake, and SQL. - Build Medallion (Bronze/Silver/Gold) layers for enterprise workloads pulling from several claims platforms and file-based feeds. - Develop and optimize production ETL/ELT pipelines with Databricks Workflows, PySpark, SQL, and Azure Data Factory ingest patterns. - Maintain dimensional data models, Delta tables, partitioning, schema evolution, and time-travel features. - Stand up Unity Catalog with role-based access, lineage, and security controls, and translate source-system data dictionaries into cataloged tables. - Establish CI/CD for notebooks and jobs, drive cost and performance tuning, and troubleshoot production data pipelines.

Requirements - 5–7 years in data engineering or cloud data platforms. - Strong hands-on expertise with Databricks, Apache Spark, advanced PySpark, and SQL. - Deep knowledge of Delta Lake and Medallion Architecture. - Production experience on Azure plus data governance, Unity Catalog, and IAM/RBAC. - Familiarity with Git, CI/CD, infrastructure-as-code, dimensional modeling, and distributed processing.

Nice to have - Azure Data Factory, Terraform, Kafka or streaming, Power BI, Databricks certifications, and prior enterprise lakehouse design.

Benefits and work setup - Pay range $100,000–$120,000 USD, with potential bonuses or sales commissions for some roles. - Medical, dental, vision, and life insurance; 401(k) with employer match; HSA and FSAs; PTO and disability leave; Employee Assistance Program.