Databricks Platform Engineer

Job Overview

We are seeking a Senior Data Platform Engineer to design, deploy, and maintain our enterprise data platform built on Databricks using Terraform. In this role, you will bridge the gap between data engineering and cloud infrastructure, automating the provisioning of Databricks workspaces, Unity Catalog, compute clusters, IAM, and automated workflows via Infrastructure as Code (IaC).

Key Responsibilities

• Automate the end-to-end deployment and lifecycle management of Databricks assets using Terraform and the Databricks Terraform Provider.
• Design and enforce governance structures, including Unity Catalog metastores, catalogs, schemas, row/column-level access controls, and service principal authentication.
• Configure, optimize, and manage Databricks compute resources (SQL Warehouses, Job Clusters, and All-Purpose Clusters) for cost efficiency and performance.
• Build CI/CD pipelines (e.g., GitHub Actions, Azure DevOps, or GitLab CI) to validate, plan, and apply Terraform configurations across Dev, Staging, and Production environments.
• Manage enterprise networking and security setups, including Private Link, VNet Peering/VPC Peering, customer-managed keys (CMK), and secret scopes.
• Partner with Data Engineers, Data Scientists, and Analytics teams to provide scalable, self-service data platform features.