Senior Data Engineer (Distributed Data Processing)
Posted Today by Xcede
Negotiable
Undetermined
Remote
Remote work , UK
Summary: The Senior Data Engineer role focuses on developing and optimizing large-scale distributed data pipelines within a data-intensive SaaS platform in a regulated industry. The position requires strong engineering skills, particularly in Python and Spark, and involves mentoring other engineers while managing complex data workflows. This is a hands-on role that emphasizes autonomy and technical leadership. It is not related to machine learning or data science.
Key Responsibilities:
- Design, build, and evolve large-scale distributed data pipelines using Spark/PySpark.
- Develop production-grade Python data workflows that implement complex business logic.
- Work with Databricks for job execution, orchestration, and optimisation.
- Own and optimise cloud-based data infrastructure (AWS preferred, Azure also relevant).
- Optimise data workloads for performance, reliability, and cost.
- Collaborate with engineers, domain specialists, and delivery teams on client-facing projects.
- Take ownership of technical initiatives and lead by example within the team.
- Support and mentor other engineers.
Key Skills:
- Proven experience as a Senior Data Engineer.
- Strong Python software engineering foundation.
- Hands-on Spark experience in production (PySpark essential).
- Real-world experience using Databricks for data pipelines (Spark depth matters most).
- Experience with large-scale or parallel data processing.
- Ownership of cloud infrastructure (AWS and/or Azure).
- Comfortable operating with senior-level autonomy and responsibility.
- Experience mentoring or supporting other engineers.
Salary (Rate): undetermined
City: undetermined
Country: UK
Working Arrangements: remote
IR35 Status: undetermined
Seniority Level: undetermined
Industry: IT
Senior Data Engineer (Distributed Data Processing)
UK (O/IR35), Belgium, Netherlands or Germany (B2B)
Fully Remote
We're looking for a Senior Data Engineer to join a data-intensive SaaS platform operating in a complex, regulated industry.
This is a hands-on senior IC role focused on distributed data processing, Spark-based pipelines, and Python-heavy engineering. You'll be working on large-scale batch data workflows that power pricing, forecasting, and operational decision-making systems.
The role requires strong engineering judgement, the ability to operate autonomously, and the confidence to mentor others while delivering under tight timelines. This is not an ML, Data Science, or GenAI role.
What You'll Be Doing
Design, build, and evolve large-scale distributed data pipelines using Spark/PySpark.
Develop production-grade Python data workflows that implement complex business logic.
Work with Databricks for job execution, orchestration, and optimisation.
Own and optimise cloud-based data infrastructure (AWS preferred, Azure also relevant).
Optimise data workloads for performance, reliability, and cost.
Collaborate with engineers, domain specialists, and delivery teams on client-facing projects.
Take ownership of technical initiatives and lead by example within the team.
Support and mentor other engineers.
Must-Have Experience
Proven experience as a Senior Data Engineer.
Strong Python software engineering foundation.
Hands-on Spark experience in production (PySpark essential).
Real-world experience using Databricks for data pipelines (Spark depth matters most).
Experience with large-scale or parallel data processing.
Ownership of cloud infrastructure (AWS and/or Azure).
Comfortable operating with senior-level autonomy and responsibility.
Experience mentoring or supporting other engineers.
Nice-to-Have Experience
Experience working with time-series data.
Background in utilities, energy, or other data-heavy regulated industries.
Exposure to streaming technologies (Kafka, event-driven systems), though the role is primarily batch-focused.