Lead Data Engineer (Databricks)

Job Title: Lead Data Engineer – Databricks (Remote, Outside IR35) Contract Type: 3-Month Contract Location: Remote (UK-based) IR35 Status: Outside IR35

Overview: We are seeking an experienced Lead Data Engineer to design, implement and validate Databricks workspaces and deliver a repeatable data migration template that standardises data product onboarding (e.g., Power BI, ShinyApps) across a modern enterprise data platform. This is a fully remote UK contract role operating Outside IR35 , suited to a senior-level consultant who can combine strong technical delivery with coaching and advisory capability. The role focuses on establishing best practice for data ingestion, transformation and serving using the Medallion Architecture, ensuring data quality, lineage and compliance with organisational standards.

Core Responsibilities

• Validate Databricks workspaces including Unity Catalog across Dev, Test and Prod environments.
• Design, implement and document best-practice data migration processes aligned to a Data-as-a-Product approach.
• Deliver end-to-end onboarding for Data Products using Databricks and Unity Catalog.
• Configure and validate Unity Catalog for metadata management, lineage tracking and data classification.
• Establish standards for ingestion, transformation and reference data handling using Medallion Architecture principles.
• Develop repeatable templates, onboarding guides and migration checklists to standardise data product integration.
• Support integration with Microsoft Purview.
• Collaborate closely with data engineers to coach, mentor and upskill internal teams.
• Operate within an Agile delivery framework, contributing to stand-ups, show-and-tells and feedback loops.
• Ensure alignment with data standards, governance and compliance requirements (including GDPR).

Essential Skills & Experience

• Proven experience operating as a Lead Data Engineer delivering enterprise-scale solutions in Databricks.
• Strong technical expertise in Databricks, including workspace design and configuration.
• Demonstrable experience implementing the Medallion Architecture for structured data transformation.
• Advanced ETL/ELT development experience using PySpark, Delta Lake and Databricks SQL.
• Practical experience with Unity Catalog for governance, metadata management and lineage tracking.
• Strong knowledge of Azure Data Platform components, including Azure Data Lake, networking and security controls.
• Experience integrating with Microsoft Purview (desirable).
• Familiarity with Microsoft Fabric (desirable).
• Strong understanding of GDPR, data standards and compliance frameworks.
• Excellent documentation skills – able to produce best practice guides, templates and onboarding frameworks.
• Lead-level capability with the ability to coach, mentor and advise client teams.
• Excellent communication and stakeholder engagement skills – upskilling and knowledge transfer are core to the role.

Key Characteristics

• Strategic thinker with strong hands-on delivery capability.
• Comfortable operating autonomously in a remote environment.
• Confident engaging with Azure, Security, Data Engineering teams and senior stakeholders.
• Focused on creating scalable, repeatable and governance-aligned solutions.

Contract Details

• Duration: 3 Months
• Location: Remote (UK-based)
• IR35 Status: Outside IR35
• Start: ASAP

This role is ideally suited to an experienced Databricks consultant who thrives in a hands-on leadership position while enabling and uplifting internal engineering teams.