Lead Data Engineer
Posted 2 days ago by Pontoon Solutions
£460 Per day
Inside
Hybrid
Warwick, England, United Kingdom
Summary: The Lead Data Engineer role in Warwick, UK, involves leading the engineering efforts for a Data & Insights platform built on Azure, focusing on Databricks and legacy Synapse technologies. The position requires overseeing data ingestion, ensuring proper configuration and documentation, and providing technical guidance to support engineers. The role is hybrid, with occasional onsite meetings, and is expected to last for six months with possible extensions. The successful candidate will lead a small team of Data Engineers and enhance the existing monitoring and control framework.
Key Responsibilities:
- Design and delivery of modern Azure and Databricks Data platform aligning to agreed technical designs and best practice.
- Responsible for data engineering across all layers of the data platform.
- Lead work on related DevOps - branch supervision, merging.
- In-depth Databricks DBA responsibilities.
- Promotion of data pipelines and associated notebooks through various environments (development, fof, production).
- Correct configuration and documentation of Databricks.
- Enhancing existing monitoring & control framework.
- Provide technical leadership & coaching for a small group of Data Engineers.
Key Skills:
- Extensive “hands on” Databricks & Synapse experience.
- Extensive Azure Dev Ops experience.
- Ideally SQL and PySpark in a Databricks environment.
- Experience of using Databricks in a federated cloud environment.
- Leadership of small engineering teams.
- Configuration management and documentation.
- Certified Data Bricks Professional Engineer.
Salary (Rate): £460 daily
City: Warwick
Country: United Kingdom
Working Arrangements: hybrid
IR35 Status: inside IR35
Seniority Level: Mid-Level
Industry: IT
Title: Lead Data Engineer
Location: Warwick, UK
Duration: 06 Months Possible extension
Work Type: Hybrid
Pay: £350 to £460 per day Umbrella (inside IR35)
Background
Client is is maturing and embedding its existing Data & Insights platform, which is on the Azure stack. We have a combination of strategic Databricks and legacy Synapse engineering. We have an existing programme team which requires a skilled individual to lead the engineering resource. This person will need to embed our Databricks architecture, ensure correct configuration and documentation, whilst also overseeing the ingestion of new data sources. It is a live platform, and the role may also need to provide technical guidance to support engineers to ensure any production issues are understood and resolved efficiently. This is a UK based role. The team predominantly work across the UK (with some off-shore team members). Occasional onsite meeting attendance.
Responsibilities
- Design and delivery of modern Azure and Databricks Data platform aligning to agreed technical designs and best practice.
- Responsible for data engineering across all layers of our data platform.
- There is currently a mix of legacy synapse and strategic Databricks.
- Responsible for leading work on related DevOps - branch supervision, merging
- In depth Databricks DBA
- Responsible for promotion of data pipelines and associated notebooks through the various environments (development, fof, production)
- Responsible for the correct configuration and documentation of Databricks
- Enhancing existing monitoring & control framework
- Provide technical leadership & coaching for a small group of Data Engineers
Required skills:
- Extensive “ hands on” Databricks & Synapse experience
- Extensive Azure Dev Ops experience
- Ideally SQL and PySpark in a Databricks environment
- Experience of using Databricks in a federated cloud environment
- Leadership of small engineering teams
- Configuration management and documentation
- Certified Data Bricks Professional Engineer