Lead Databricks Unity Catalog Engineer

Job Summary

We are looking for an experienced Lead or Developer with strong expertise in Unity Catalog, data governance, and enterprise data platform management.

The ideal candidate should have hands-on experience with Unity Catalog, access control, metadata management, data lineage, and governance within a Databricks environment.

The candidate will lead data governance initiatives, manage platform integrations, and collaborate with data engineering, DevOps, security, and business teams to establish scalable and secure data management practices.

Experience with Azure DevOps, GitHub, Atlassian, and Jira is required.

Experience in Wealth Management or Financial Services is highly preferred.

Key Responsibilities

• Lead the implementation, administration, and management of Databricks Unity Catalog.
• Establish and manage data governance policies, access controls, permissions, and security standards.
• Implement and maintain data lineage, metadata management, and data discovery processes.
• Manage Unity Catalog integration with Databricks workspaces and enterprise data platforms.
• Work with Azure DevOps and GitHub to manage CI/CD pipelines, version control, and deployment processes.
• Establish and manage development workflows using GitHub and Azure DevOps.
• Utilize Jira and Atlassian tools for project tracking, sprint planning, task management, and team collaboration.
• Collaborate with Data Engineering, Cloud, Security, and Architecture teams to implement governance standards.
• Define best practices for data access management, auditability, and compliance.
• Troubleshoot platform, deployment, access, and integration-related issues.
• Lead technical discussions, mentor team members, and coordinate with stakeholders.
• Ensure data governance practices align with enterprise security and regulatory requirements.

Required Qualifications

• 8+ years of experience in Data Engineering, Data Platform Management, or related technical roles.
• Strong hands-on experience with Databricks Unity Catalog.
• In-depth understanding of data governance, metadata management, data lineage, and access controls.
• Experience with role-based access control (RBAC), permissions, and data security.
• Hands-on experience with Azure DevOps and GitHub.
• Experience managing CI/CD pipelines, Git repositories, and deployment workflows.
• Working experience with Atlassian tools, particularly Jira.
• Strong understanding of cloud-based data platforms and enterprise data architecture.
• Excellent leadership, communication, problem-solving, and stakeholder management skills.

Preferred Qualifications

• Experience with Azure Databricks and Microsoft Azure cloud services.
• Knowledge of Delta Lake, Apache Spark, and lakehouse architecture.
• Experience implementing enterprise data governance frameworks.
• Understanding of financial services, wealth management, and regulatory compliance requirements.
• Experience leading technical teams and enterprise data platform initiatives.