Senior Data Engineer
Posted Today by MRJ Recruitment
Negotiable
Undetermined
Remote
United Kingdom
Summary: The Senior Data Engineer role involves leading the modernization of a legacy data warehouse by migrating to an AWS Databricks platform. The position requires building a Common Data Model to support analytics and performance reporting, along with designing data pipelines and ensuring data quality. The role is remote and is expected to last for an initial period of 5 months. The ideal candidate will have extensive experience with AWS Databricks and data engineering best practices.
Key Responsibilities:
- Lead discovery activities to understand the current data warehouse, dependencies, data flows and reporting needs
- Design and build modern data pipelines and data models on AWS and Databricks to support analytics and downstream consumption
- Migrate and re-platform data from legacy systems into the new cloud environment, ensuring accuracy, performance and reliability
- Collaborate with stakeholders to capture requirements and translate them into practical, scalable engineering solutions
- Drive best practice across data engineering, including quality, monitoring, performance optimisation and deployment processes
- Support deployment activities, release planning and operational readiness as the new platform goes live
Key Skills:
- Extensive experience of using AWS Databricks
- Experienced in building scalable data pipelines using SQL, Python and Pyspark
- Hands on experience of designing and operating Delta Live Table pipelines in both streaming and rejiggered modes
- Strong experience of selecting the right Databricks compute for different workloads
- Has worked extensively with the Unity Catalog to centralise governance across Databricks workloads
- Can diagnose and optimise slow queries to guide and improve performance of existing structure/design
- Experience of using Databricks Asset Bundles to define, package and deploy resources as code
Salary (Rate): undetermined
City: undetermined
Country: United Kingdom
Working Arrangements: remote
IR35 Status: undetermined
Seniority Level: Senior
Industry: IT
Role: Senior Data Engineer
Length: 5 months initially
Location: Remote UK
Start date: ASAP
About the Role
Our client is currently undertaking a key data programme of work modernising their legacy data warehouse, migrating to a modern AWS Databricks platform. As part of this project, we are looking for an experienced Senior Data Engineer to help us build a Common Data Model in AWS Databricks to support analytics, performance reporting and downstream consumption across the platform for the company. Work will include modelling, data ingestion, transformation between the medallion layers and documentation to enable consistent, scalable use of our data across our domains.
- Key Responsibilities
- Lead discovery activities to understand the current data warehouse, dependencies, data flows and reporting needs
- Design and build modern data pipelines and data models on AWS and Databricks to support analytics and downstream consumption
- Migrate and re-platform data from legacy systems into the new cloud environment, ensuring accuracy, performance and reliability
- Collaborate with stakeholders to capture requirements and translate them into practical, scalable engineering solutions
- Drive best practice across data engineering, including quality, monitoring, performance optimisation and deployment processes
- Support deployment activities, release planning and operational readiness as the new platform goes live
Person Requirements
Extensive experience of using AWS Databricks
Experienced in building scalable data pipelines using SQL, Python and Pyspark
Hands on experience of designing and operating Delta Live Table pipelines (now called Lakeflow Declarative Spark Pipelines) in both streaming and rejiggered modes
Strong experience of selecting the right Databricks compute for different workloads
Has worked extensively with the Unity Catalog to centralise governance across Databricks workloads
Can diagnose and optimise slow queries to guide and improve performance of existing structure/design
Experience of using Databricks Asset Bundles to define, package and deploy resources as code
Next steps: If you’re a Senior Data Engineer ready to lead a move to Databricks and want a role where your work will shape a modern data platform, we’d like to hear from you. Shortlisting will take place on the this week with a quick turnaround expected for interviews and start.