Senior / Lead Data Engineer
Posted 1 day ago by The ProEx Group
Negotiable
Outside
Hybrid
London Area, United Kingdom
Summary: The Senior / Lead Data Engineer role is focused on designing, building, and governing enterprise-grade data platforms that facilitate analytics, AI, and operational decision-making. This position requires a blend of hands-on engineering, technical architecture, and stakeholder leadership, working across business and technology teams to deliver scalable and secure data products. The role demands extensive experience in data engineering and analytics, particularly within complex environments. Candidates will be responsible for leading data engineering teams and ensuring the delivery of high-quality data solutions.
Key Responsibilities:
- Design, build, and maintain end-to-end data integration pipelines across cloud platforms (primarily Azure).
- Develop scalable ELT/ETL pipelines to support analytics, AI/ML, BI, and operational systems.
- Implement data quality monitoring, lineage, validation, and governance.
- Build and optimise data models for analytical and AI workloads.
- Integrate data from complex ecosystems including APIs, geospatial services, and third-party vendors.
- Develop production-ready pipelines with CI/CD using Azure DevOps.
- Own high-level and low-level technical designs across data platforms and APIs.
- Perform technical impact assessments for new data sources and features.
- Define scalable and secure architectures considering performance and regulatory requirements.
- Collaborate with data scientists and business leaders to define KPIs and data products.
- Lead, coach, and mentor data engineers.
- Act as a trusted technical advisor to senior stakeholders.
Key Skills:
- 10+ years in data engineering, analytics, or data architecture roles.
- Proven delivery across enterprise-scale data platforms.
- Advanced SQL, Python, and PySpark skills.
- Experience with Azure cloud services (ADF, Databricks, Synapse).
- Strong knowledge of data modelling and ML data pipelines.
- Exceptional stakeholder management and communication skills.
- Proven team leadership and coaching abilities.
Salary (Rate): undetermined
City: London
Country: United Kingdom
Working Arrangements: hybrid
IR35 Status: outside IR35
Seniority Level: Senior
Industry: IT
Senior / Lead Data Engineer
Location: London/Hybrid
Reports to: Delivery Lead
Level: Senior Manager / Lead Individual Contributor
Contract: Contract – Outside IR35
Please note – Due to the number of applicants, only those who meet or exceed both the technical and business skills will be considered for the next phase. Only successful candidates will be contacted by our Talent team.
Role Purpose
We are seeking an exceptional Senior / Lead Data Engineer to design, build, and govern enterprise-grade data platforms that power analytics, AI, and operational decision-making. This role blends hands-on engineering , technical architecture , and stakeholder leadership . You will work across business and technology teams to translate complex requirements into scalable, secure, and high-quality data products.
Key Responsibilities
- Data Engineering & Platform Delivery
- Design, build, and maintain end-to-end data integration pipelines across cloud platforms (primarily Azure) using tools such as Azure Data Factory, Databricks, PySpark, SQL, and Python .
- Develop scalable ELT/ETL pipelines to support analytics, AI/ML, BI, and operational systems.
- Implement data quality monitoring, lineage, validation, and governance to ensure trusted data products.
- Build and optimise data models for analytical and AI workloads (e.g., Azure SQL, Synapse, PostgreSQL, Databricks Delta).
- Integrate data from complex ecosystems including APIs, geospatial services, SaaS platforms, on-prem systems, and third-party vendors .
- Develop production-ready pipelines with CI/CD using Azure DevOps, Git, and release pipelines .
- Technical Architecture & Design
- Own high-level and low-level technical designs across data platforms, APIs, and integration services.
- Perform technical impact assessments for new data sources, features, and business requirements.
- Define scalable and secure architectures considering performance, cost, resilience, and regulatory requirements.
- Produce options papers, technical roadmaps, and cost-of-ownership models to support investment decisions.
- Ensure operational readiness by defining service wrap, monitoring, incident management, and interface agreements with system owners.
- Analytics, AI & Business Enablement
- Collaborate with data scientists, analysts, product owners, and business leaders to define KPIs and data products that drive measurable value.
- Support feature engineering, model data pipelines, and AI use cases (e.g., demand forecasting, optimisation, clustering, ML pipelines).
- Enable business intelligence platforms (e.g., Power BI, Tableau ) with trusted, well-modelled datasets.
- Leadership & Stakeholder Management
- Lead, coach, and mentor data engineers (including graduates and diverse teams).
- Act as a trusted technical advisor to senior stakeholders and clients.
- Translate business needs into technical solutions and clearly communicate risks, trade-offs, and recommendations.
- Support sales, pre-sales, and roadmap definition for future data initiatives.
Required Experience & Skills
Experience
- 10+ years in data engineering, analytics, or data architecture roles.
- Proven delivery across enterprise-scale data platforms in regulated or complex environments (e.g., public sector, infrastructure, utilities, retail, FMCG).
- Experience operating in consulting or matrixed organisations with multiple stakeholders.
Technical Skills
- Advanced: SQL, Python, PySpark
- Cloud: Azure (ADF, Databricks, Synapse, Azure SQL, Blob Storage, Power Automate)
- Data Platforms: Databricks, relational & analytical data stores, API integration
- DevOps: Azure DevOps, CI/CD pipelines, source control
- Analytics & AI: Data modelling, feature engineering, ML data pipelines
- Integration: APIs, geospatial data, third-party systems, event-based pipelines
- Architecture & Governance
- Data architecture and platform design
- Data quality, lineage, and governance frameworks
- Security, GDPR, and data compliance considerations
Soft Skills
- Exceptional stakeholder management and communication
- Strong risk management and problem-solving
- Proven team leadership and coaching
- Ability to bridge business and technical teams
What Success Looks Like
- Trusted, scalable data platforms supporting AI, analytics, and operations
- High-quality, governed, and auditable datasets
- Strong adoption by business users
- A high-performing, upskilled data engineering team
- Clear technical roadmaps aligned to business strategy
How do you apply?
Contact us at talent@theproexgroup.com and include the following:
- Curriculum Vitae / Resume
- Current Employment Position (Including notice period)
- Contact Details
- Remuneration Expectation
- Optional Cover Letter
Be sure to follow our LinkedIn profile for more exciting global opportunities across a range of industries and sectors