Senior Data Engineer
Posted 2 weeks ago by Fruition Group
Negotiable
Outside
Remote
England, UK
Summary: This role is for a Senior Data Engineer focused on a high-volume data transformation programme, primarily involving AWS-based migration projects. The position requires collaboration with an engineering team to tackle complex data challenges, particularly in ETL processes. The contract is outside IR35 and offers a competitive market rate, with a duration of 6 months and potential for extension.
Key Responsibilities:
- Design, build, and optimise large-scale ETL processes using AWS Glue.
- Repartition and transform existing data pipelines.
- Develop scalable logic in Glue Jobs to accommodate massive data volumes, ranging from tens to hundreds of terabytes.
- Contribute to data architecture decisions by identifying non-obvious segmentation strategies and designing new formats for extraction and storage.
- Ensure reliable delivery of high-volume transaction data to S3, handling millions of files across a dynamic architecture.
Key Skills:
- Proven experience designing and delivering large-scale ETL pipelines, preferably from relational databases.
- Strong understanding of relational data models and how they map into distributed data environments.
- Hands-on expertise with AWS Glue, particularly in coding and embedding business logic into Glue Jobs.
- Comfortable managing large data sets and optimising performance at scale in cloud environments.
- Able to troubleshoot data flow and transform logic efficiently in a complex pipeline.
- Experience working as a Lead/Principal Data Engineer, or Entry-Level Architect in high-volume data environments.
Salary (Rate): undetermined
City: undetermined
Country: UK
Working Arrangements: remote
IR35 Status: outside IR35
Seniority Level: Senior
Industry: IT
Job Title: Senior Data Engineer
Location: Remote (UK-based)
Contract: Outside IR35, Competitive Market Rate
Duration: 6 months with strong chance of extension
Why Apply?
This is a unique opportunity to play a key role in a high volume data transformation programme. You'll work with a collaborative engineering team on a large scale AWS based migration project, solving complex data partitioning and performance challenges.
Responsibilities
- Design, build, and optimise large-scale ETL processes using AWS Glue.
- Repartition and transform existing data pipelines.
- Develop scalable logic in Glue Jobs to accommodate massive data volumes, ranging from tens to hundreds of terabytes.
- Contribute to data architecture decisions by identifying non-obvious segmentation strategies and designing new formats for extraction and storage.
- Ensure reliable delivery of high-volume transaction data to S3, handling millions of files across a dynamic architecture.
Requirements
- Proven experience designing and delivering large-scale ETL pipelines, preferably from relational databases.
- Strong understanding of relational data models and how they map into distributed data environments.
- Hands-on expertise with AWS Glue, particularly in coding and embedding business logic into Glue Jobs.
- Comfortable managing large data sets and optimising performance at scale in cloud environments.
- Able to troubleshoot data flow and transform logic efficiently in a complex pipeline.
- Experience working as a Lead/Principal Data Engineer, or Entry-Level Architect in high-volume data environments.
We are an equal opportunities employer and welcome applications from all suitably qualified persons regardless of their race, sex, disability, religion/belief, sexual orientation or age.