Senior Data Engineer
Posted 1 day ago by Primus
£750 Per day
Undetermined
Remote
Birmingham, England, United Kingdom
Summary: The Lead Data Engineer will oversee a team of data engineers and manage the enterprise Data & Insights Platform, which is crucial for data management and reporting in asset management, finance, and investment. This role involves designing and optimizing data pipelines, migrating from Synapse to Databricks, and ensuring best practices in engineering. The successful candidate will also engage in hands-on engineering work while leading the team and collaborating with stakeholders. The position is remote with occasional travel to Birmingham and offers a competitive daily rate.
Key Responsibilities:
- Lead the Data Engineering team (c.5 people), setting technical direction.
- Design, build, and optimise data pipelines end-to-end.
- Support migration from Synapse to Databricks.
- Ensure best practice in documentation, configuration management, and use of Unity Catalog.
- Act as a hands-on engineer (SQL & PySpark) while also leading delivery.
- Collaborate closely with programme management and wider stakeholders.
Key Skills:
- Extensive Databricks engineering experience.
- Strong PySQL & PySpark in a Databricks environment.
- Hands-on pipeline development and data modelling experience.
- Capable of reviewing code and enforcing standards, naming conventions etc.
- Experience of optimisation / cost control.
- Track record of technical leadership within small/mid-sized teams.
- Excellent communication skills with the ability to explain complex solutions to non-technical stakeholders.
Salary (Rate): £750 daily
City: Birmingham
Country: United Kingdom
Working Arrangements: remote
IR35 Status: undetermined
Seniority Level: undetermined
Industry: IT
Lead Data Engineer (Databricks) Remote (occasional travel to Birmingham) £650 - £750 per day depending on experience 6 months initially (extension likely) ASAP start (2-week onboarding)
About the Role
We are seeking a Lead Data Engineer to join our client and play a key role in their enterprise Data & Insights Platform. This platform is central to how the business manages and reports on critical data assets across asset management, finance, and investment. The environment is Azure-based, with a hybrid of legacy Synapse pipelines and new Databricks solutions. The successful candidate will take ownership of engineering practices, pipeline development, and lead a small, mixed team of data engineers (including offshore).
Key Responsibilities
- Lead the Data Engineering team (c.5 people), setting technical direction.
- Design, build, and optimise data pipelines end-to-end
- Support migration from Synapse to Databricks.
- Ensure best practice in documentation, configuration management, and use of Unity Catalog.
- Act as a hands-on engineer (SQL & PySpark) while also leading delivery.
- Collaborate closely with programme management and wider stakeholders.
Essential Skills
- Extensive Databricks engineering experience
- Strong PySQL & PySpark in a Databricks environment.
- Hands-on pipeline development and data modelling experience.
- Capable of reviewing code and enforcing standards, naming conventions etc
- Experience of optimisation / cost control
- Track record of technical leadership within small/mid-sized teams.
- Excellent communication skills with the ability to explain complex solutions to non-technical stakeholders.
Desirable
- Previous experience in regulated industries (utilities, energy, financial services).
- Databricks Certified Professional
- Strong DevOps knowledge within data engineering environments.
Why Join?
- Influence and lead a strategic enterprise data platform.
- Long-term project with funding in place and strong likelihood of extension.
- Collaborative environment, flexible working, and impactful role.
To apply, please click apply now.