Key Responsibilities
- Design, build and maintain data pipelines using Azure Databricks
- Develop production-grade data engineering solutions using PySpark and SQL
- Implement and maintain CI/CD pipelines for Databricks using Azure DevOps
- Build automated deployment processes using Azure DevOps YAML pipelines, Git repositories and branching strategies
- Deploy Databricks notebooks and pipelines across environments using automated release processes
- Work with Databricks Asset Bundles, Databricks Repos and environment promotion
- Build and maintain Databricks-native dashboards, including Databricks SQL/AI-BI dashboards
- Develop data solutions using Azure Data Lake Storage (ADLS)/Blob Storage
- Work with Azure Functions and Azure Container Apps as part of wider data engineering solutions
- Apply sound data modelling principles, including medallion architecture and/or dimensional modelling
- Build robust, scalable and maintainable end-to-end data pipelines
- Troubleshoot and optimise existing data pipelines and Databricks workloads
- Work closely with technical stakeholders to deliver against an urgent project timeline
Skills & Experience
- 4-7 years' commercial experience in Data Engineering
- Strong hands-on experience with Azure Databricks
- Strong PySpark and SQL skills
- Proven experience building production data pipelines in Databricks
- Strong experience with Azure DevOps
- Experience creating Azure DevOps CI/CD pipelines using YAML
- Experience with Git repositories, branching and automated deployment processes
- Experience deploying Databricks workloads through CI/CD rather than relying on manual notebook execution
- Experience with Databricks Asset Bundles and/or Databricks Repos
- Hands-on experience building Databricks-native dashboards/Databricks SQL or AI-BI dashboards
- Strong experience with ADLS/Azure Blob Storage
- Experience with Azure Functions and/or Azure Container Apps
- Good understanding of data modelling and data engineering principles
- Strong written and spoken English
- Ability to work with sufficient overlap with UK working hours
- Immediate or very short-term availability
Desirable
- Experience with Delta Lake
- Experience implementing medallion architecture
- Experience with Azure Key Vault and cloud security
- Experience with infrastructure-as-code
- Experience optimising Databricks performance and cloud costs
- Previous experience working on short-term, fast-paced delivery projects
This role is particularly suited to a hands-on Data Engineer, rather than someone whose experience is primarily architectural or managerial.
We are looking for someone who can take a requirement and independently build the solution from data ingestion through to transformation, deployment and dashboarding.
Candidates whose Azure experience is primarily ADF/Synapse with limited hands-on Databricks experience are unlikely to be suitable.
Similarly, candidates who have only used Databricks to run notebooks manually, without experience deploying workloads through CI/CD, will not be considered a strong match.
Contract Details
- 3-month initial contract
- Immediate start required
- Offshore/remote engagement
- Competitive, cost-conscious contract rate
- UK-hours overlap
Rate:
Not specified
Location:
London
IR35 Status:
Not specified
Remote Status:
Remote
Industry:
Data & Analytics
Seniority Level:
Mid-Level