Lead Azure Databricks Engineer
Posted 7 days ago by eTeam Workforce Limited
£492 Per day
Inside
Hybrid
Manchester, UK
Summary: The Lead Azure Databricks Engineer will be responsible for the design, build, optimization, and governance of enterprise data platforms, specifically within the Lloyd's of London market. This role requires a combination of technical expertise and hands-on leadership to drive cloud data strategy and ensure robust solutions. The position is highly collaborative, working closely with various functions to meet regulatory and operational requirements. This is a hands-on role that demands a proactive approach to continuous improvement and technical challenges.
Key Responsibilities:
- Lead the development and optimisation of large-scale data platforms using Azure and Databricks.
- Work closely with various functions to deliver solutions aligned to Lloyd's of London regulatory and operational requirements.
- Partner with architects, SMEs, product owners, and leadership to translate business needs into scalable cloud data solutions.
- Own and resolve complex technical challenges with speed and clarity.
- Champion best-practice design patterns, data life cycle management, CI/CD, automation, and cloud-native engineering principles.
- Provide technical leadership and mentoring, setting engineering standards and uplifting capability across the team.
- Drive continuous improvement, proactively identifying opportunities to enhance platform performance, reduce cost, and improve resilience.
- Oversee end-to-end data engineering delivery, ensuring high-quality pipelines, reliability, observability, and performance.
Key Skills:
- Expert-level engineering experience across Azure Data Services and Databricks in enterprise-scale environments.
- Deep understanding of Delta Lake, Medallion architecture, distributed compute, Lakehouse patterns, data modelling and optimisation.
- Strong Python, PySpark and Spark SQL skills, plus experience implementing CI/CD for data solutions.
- Solid understanding of data governance, lineage, access controls, FinOps, and secure cloud engineering practices.
- Outstanding communication skills, able to interface confidently with senior stakeholders.
- Delivery-focused mindset with a proven track record of ownership and accountability.
- Experience working in fast-paced, high-expectation environments with tight deadlines.
Salary (Rate): £492/Day
City: Manchester
Country: UK
Working Arrangements: hybrid
IR35 Status: inside IR35
Seniority Level: undetermined
Industry: IT
We are a Global Recruitment specialist that provides support to the clients across EMEA, APAC, US and Canada. We have an excellent job opportunity for you.
Role Title: Lead Azure Databricks Engineer
Location: Manchester
Duration: 31/12/2026
Work setup: 2 days onsite/week
Rate: £492/Day - Inside IR35
MUST BE PAYE THROUGH UMBRELLA
Role Description:
We are seeking an exceptional Lead Azure & Databricks Engineer to drive the design, build, optimisation, and governance of our enterprise data platforms. The ideal candidate combines deep technical expertise with hands-on leadership, excels in complex and highly regulated environments, and has proven delivery experience within the Lloyd's of London market. You will play a pivotal role in shaping our cloud data strategy, enabling advanced analytics, and ensuring robust, scalable and secure solutions across our organisation. This is 100% hands on role.
Key Responsibilities:
Lead the development, and optimisation of large-scale data platforms using Azure (ADF, Data Lake, Key Vault, Azure Functions) and Databricks (Delta Lake, PySpark, Unity Catalog).
Work closely with underwriting, actuarial, delegated authority, bordereaux, exposure management, Reinsurance, Finance Solvency II and risk functions to deliver solutions aligned to Lloyd's of London regulatory and operational requirements, including compliance, auditability, and data lineage.
Partner with architects, SMEs, product owners, and leadership to translate business needs into scalable cloud data solutions.
Own and resolve complex technical challenges with speed and clarity-someone who consistently gets the job done.
Champion best-practice design patterns, data life cycle management, CI/CD, automation, and cloud-native engineering principles.
Provide technical leadership and mentoring, setting engineering standards and uplifting capability across the team.
Drive continuous improvement, proactively identifying opportunities to enhance platform performance, reduce cost, and improve resilience.
Oversee end-to-end data engineering delivery, ensuring high-quality pipelines, reliability, observability, and performance.
Required Experience & Skills:
Expert-level engineering experience across Azure Data Services and Databricks in enterprise-scale environments.
Deep understanding of Delta Lake, Medallion architecture, distributed compute, Lakehouse patterns, data modelling and optimisation.
Strong Python, PySpark and Spark SQL skills, plus experience implementing CI/CD for data solutions (Azure DevOps, Branching and merging).
Solid understanding of data governance, lineage, access controls, FinOps, and secure cloud engineering practices.
Outstanding communication skills-able to interface confidently with senior stakeholders, articulate complex concepts clearly, and lead cross-functional discussions.
Delivery-focused mindset with a proven track record of ownership, accountability, and seeing initiatives through to completion.
Experience working in fast-paced, high-expectation environments with tight deadlines and evolving requirements.
If you are interested in this position and would like to learn more, please send through your CV and we will get in touch with you as soon as possible. Please note, candidates are often Shortlisted within 48 hours.