SC Cleared Data Engineer

Key Responsibilities

• Design, build and optimise Azure data pipelines using Azure Data Factory and Azure Databricks
• Develop data ingestion solutions using Change Data Capture and Managed File Transfer
• Build canonical data models, conformed dimensions and fact tables
• Deliver curated Gold-layer datasets for reporting, analytics and AI use cases
• Automate data validation and quality assurance processes, replacing manual checks
• Configure Unity Catalog, clusters, pipelines and data contracts
• Contribute to data governance, security, lineage and platform design
• Collaborate on solution design, code reviews, testing and defect resolution

Essential Skills & Experience

• Current active SC Clearance (mandatory)
• Strong commercial experience with Azure Databricks, Azure Data Factory and Azure data engineering
• Advanced Python (including Pandas) and SQL
• Hands-on experience with Unity Catalog
• Experience designing data warehouses, including slowly changing dimensions (SCD) and dimensional modelling
• Knowledge of CDC, data ingestion patterns and governed data platforms
• Understanding of data governance, security and lineage
• Experience within regulated or public sector environments is advantageous
• Familiarity with Power BI is beneficial but not essential