Senior Energy Markets Data Engineer
Posted 4 days ago by The ProEx Group
Negotiable
Outside
Hybrid
London Area, United Kingdom
Summary: The Senior Energy Markets Data Engineer role involves designing, building, and governing enterprise-grade data platforms that facilitate analytics and operational decision-making. This position requires a blend of hands-on engineering, technical architecture, and stakeholder leadership, specifically within the energy markets sector. The successful candidate will work closely with business and technology teams to create scalable and secure data products. Candidates must have recent experience in energy markets and a strong background in data engineering and analytics.
Key Responsibilities:
- Design, build, and maintain end-to-end data integration pipelines across cloud platforms (primarily Azure).
- Develop scalable ELT/ETL pipelines to support analytics, AI/ML, BI, and operational systems.
- Implement data quality monitoring, lineage, validation, and governance.
- Build and optimise data models for analytical and AI workloads.
- Integrate data from complex ecosystems including APIs, geospatial services, and third-party vendors.
- Develop production-ready pipelines with CI/CD using Azure DevOps.
- Own high-level and low-level technical designs across data platforms and APIs.
- Perform technical impact assessments for new data sources and business requirements.
- Define scalable and secure architectures considering performance and regulatory requirements.
- Collaborate with data scientists and business leaders to define KPIs and data products.
- Lead, coach, and mentor data engineers.
- Act as a trusted technical advisor to senior stakeholders and clients.
Key Skills:
- Recent Energy Markets experience.
- 10+ years in data engineering, analytics, or data architecture roles.
- Proven delivery across enterprise-scale data platforms.
- Advanced SQL, Python, and PySpark skills.
- Experience with Azure cloud services (ADF, Databricks, Synapse).
- Knowledge of data modelling, feature engineering, and ML data pipelines.
- Exceptional stakeholder management and communication skills.
- Strong risk management and problem-solving abilities.
- Proven team leadership and coaching experience.
Salary (Rate): undetermined
City: London
Country: United Kingdom
Working Arrangements: hybrid
IR35 Status: outside IR35
Seniority Level: Senior
Industry: IT
Senior Energy Markets Data Engineer
Location: London/Hybrid (3/4 Days in London Office)
Reports to: Delivery Lead
Level: Senior Manager / Lead Individual Contributor
Contract: Contract – Outside IR35
Please note – only candidates with recent Energy Markets experience will be considered for this role.
Role Purpose
We are seeking an exceptional Senior / Lead Data Engineer to design, build, and govern enterprise-grade data platforms that power analytics, AI, and operational decision-making. This role blends hands-on engineering , technical architecture , and stakeholder leadership . You will work across business and technology teams to translate complex requirements into scalable, secure, and high-quality data products.
Key Responsibilities
- Data Engineering & Platform Delivery
- Design, build, and maintain end-to-end data integration pipelines across cloud platforms (primarily Azure) using tools such as Azure Data Factory, Databricks, PySpark, SQL, and Python .
- Develop scalable ELT/ETL pipelines to support analytics, AI/ML, BI, and operational systems.
- Implement data quality monitoring, lineage, validation, and governance to ensure trusted data products.
- Build and optimise data models for analytical and AI workloads (e.g., Azure SQL, Synapse, PostgreSQL, Databricks Delta).
- Integrate data from complex ecosystems including APIs, geospatial services, SaaS platforms, on-prem systems, and third-party vendors .
- Develop production-ready pipelines with CI/CD using Azure DevOps, Git, and release pipelines .
- Technical Architecture & Design
- Own high-level and low-level technical designs across data platforms, APIs, and integration services.
- Perform technical impact assessments for new data sources, features, and business requirements.
- Define scalable and secure architectures considering performance, cost, resilience, and regulatory requirements.
- Produce options papers, technical roadmaps, and cost-of-ownership models to support investment decisions.
- Ensure operational readiness by defining service wrap, monitoring, incident management, and interface agreements with system owners.
- Analytics, AI & Business Enablement
- Collaborate with data scientists, analysts, product owners, and business leaders to define KPIs and data products that drive measurable value.
- Support feature engineering, model data pipelines, and AI use cases (e.g., demand forecasting, optimisation, clustering, ML pipelines).
- Enable business intelligence platforms (e.g., Power BI, Tableau ) with trusted, well-modelled datasets.
- Leadership & Stakeholder Management
- Lead, coach, and mentor data engineers (including graduates and diverse teams).
- Act as a trusted technical advisor to senior stakeholders and clients.
- Translate business needs into technical solutions and clearly communicate risks, trade-offs, and recommendations.
- Support sales, pre-sales, and roadmap definition for future data initiatives.
Required Experience & Skills
Experience
- Recent Energy Markets experience is a must
- 10+ years in data engineering, analytics, or data architecture roles.
- Proven delivery across enterprise-scale data platforms in regulated or complex environments (e.g., public sector, infrastructure, utilities, retail, FMCG).
- Experience operating in consulting or matrixed organisations with multiple stakeholders.
Technical Skills
- Advanced: SQL, Python, PySpark
- Cloud: Azure (ADF, Databricks, Synapse, Azure SQL, Blob Storage, Power Automate)
- Data Platforms: Databricks, relational & analytical data stores, API integration
- DevOps: Azure DevOps, CI/CD pipelines, source control
- Analytics & AI: Data modelling, feature engineering, ML data pipelines
- Integration: APIs, geospatial data, third-party systems, event-based pipelines
- Architecture & Governance
- Data architecture and platform design
- Data quality, lineage, and governance frameworks
- Security, GDPR, and data compliance considerations
Soft Skills
- Exceptional stakeholder management and written/verbal communication
- Strong risk management and problem-solving
- Proven team leadership and coaching
- Ability to bridge business and technical teams
Be sure to follow our LinkedIn profile for more exciting global opportunities across a range of industries and sectors