Data Engineering Lead (Data Acquisition)

Data Engineering Lead (Data Acquisition)

Posted 3 days ago by Eagle cliff

Apply
Negotiable
Undetermined
Undetermined
London, Waterloo
Apply
Application Programming Interface (API) Azure Databricks Bloomberg Terminal Continuous Integration and Continuous Delivery Data Acquisition Databricks Data Engineering Data Ingestion Data Management Finance Financial Market Management Market Data Metadata Microsoft Azure Python (Programming Language) SAP Applications Software Coding SQL (Programming Language) Storage Systems Team Leadership

Summary: Exciting opportunity for a Data Engineering Lead (Data Acquisition) to join a global Energy company undergoing transformation from a traditional Oil & Gas enterprise to a future-focused Energy leader. The role involves leading a data acquisition function within the Energy Trading & Supply organization, focusing on ingestion strategy and pipeline delivery on Azure, while managing a team of data engineers. This position is critical in supporting the new Power Trading desk and addressing the evolving demands of the Energy sector. The ideal candidate will have extensive experience in data engineering and team leadership in complex environments.

Key Responsibilities:

  • Own and define the data ingestion strategy; design and build scalable pipelines on Azure (ADLS, ADF, Databricks, Synapse, Event Hubs/Kafka) with robust observability and SLAs.
  • Translate requirements from trading, risk, operations and finance into a prioritised engineering backlog, working closely with architects and product owners.
  • Serve as the domain’s Data Manager, stewarding data definitions, lineage, quality, metadata and access policies.
  • Lead, coach and develop a team of data engineers, driving high standards in coding, design and production operations.
  • Ensure data platforms are resilient, secure and compliant with relevant trading regulations (REMIT/EMIR).

Key Skills:

  • 10+ years in data engineering, with 3-5 years leading teams in complex, real-time environments.
  • Strong hands-on expertise in Azure data services and Python, SQL, Spark/PySpark.
  • Solid understanding of streaming and batch ingestion patterns, schema management and API integration.
  • Proven ability to lead engineering teams in an Agile/DevOps environment with CI/CD practices.
  • Experience with energy trading systems (ETRM/Endur), market data platforms (ICE, Platts, Bloomberg) and SAP is advantageous.
  • Degree in Computer Science/Engineering or equivalent; Azure or Databricks certifications desirable.

Salary (Rate): undetermined

City: London

Country: UK

Working Arrangements: undetermined

IR35 Status: undetermined

Seniority Level: undetermined

Industry: Energy

Detailed Description From Employer:

Exciting opportunity for a Data Engineering Lead (Data Acquisition) to join this global Energy company.

This is an exciting time to join this world-leading Energy company as it evolves from a traditional Oil & Gas enterprise into a future-focused Energy leader. This transformation includes the establishment of a highly successful new Power Trading desk within the Energy Trading team and ventures into cutting-edge solutions that address the evolving demands of the Energy sector. These initiatives include advanced Gas infrastructure, LNG technology, effective Energy Storage systems and Green Energy solutions.

You will lead a data acquisition function within an Energy Trading & Supply organisation, owning ingestion strategy, pipeline delivery on Azure, and a team of data engineers serving Trading, Risk, Finance and Operations.

Key Responsibilities

• Own and define the data ingestion strategy; design and build scalable pipelines on Azure (ADLS, ADF, Databricks, Synapse, Event Hubs/Kafka) with robust observability and SLAs.
• Translate requirements from trading, risk, operations and finance into a prioritised engineering backlog, working closely with architects and product owners.
• Serve as the domain’s Data Manager, stewarding data definitions, lineage, quality, metadata and access policies.
• Lead, coach and develop a team of data engineers, driving high standards in coding, design and production operations.
• Ensure data platforms are resilient, secure and compliant with relevant trading regulations (REMIT/EMIR).

Experience & Skills

• 10+ years in data engineering, with 35 years leading teams in complex, real-time environments.
• Strong hands-on expertise in Azure data services and Python, SQL, Spark/PySpark.
• Solid understanding of streaming and batch ingestion patterns, schema management and API integration.
• Proven ability to lead engineering teams in an Agile/DevOps environment with CI/CD practices.
• Experience with energy trading systems (ETRM/Endur), market data platforms (ICE, Platts, Bloomberg) and SAP is advantageous.
• Degree in Computer Science/Engineering or equivalent; Azure or Databricks certifications desirable.

With a focus within Energy Trading, Oil & Gas, Financial Markets and Commodities, we offer a transparent Recruitment Service that has proven to be reliable and effective for over 40 years. We are ISO accredited and proud of our excellent TrustPilot Reviews.

Eaglecliff Ltd is acting in the capacity of an employment agency for permanent recruitment and an employment business for contractor resourcing.

Apply
Similar Jobs
Loading data...