Role: Principal Data Engineer (Azure Databricks)
Programme: Data Exchange Platform
Industry: Commodities / Energy Trading
Location: London (Hybrid)
Contract: Initial 6-month contract (long-term transformation programme)
Rate: £850-950/day (Inside IR35)
Overview
X4 Technology are partnered with a global energy trading firm on a major transformation of its Data Exchange platform - the technical data backbone supporting physical commodities trading globally.
The platform automates, reconciles and standardises transaction workflows across physical crude oil and refined petroleum products, providing a single, trusted source of data across the trading ecosystem.
Responsibilities for the Principal Data Engineer (Databricks)
- Own the end-to-end data and platform architecture, shaping scalable, secure and high-performance solutions across multiple workstreams
- Lead the design and delivery of the Databricks data platform, covering batch and real-time ingestion, streaming, data modelling and integration across Azure
- Set technical direction and engineering standards, driving best practice across performance, scalability, governance, security, testing and cost optimisation
- Act as the technical focal point across Architects, Engineers, Programme Managers, Business Analysts and Trading stakeholders
- Identify and implement opportunities for AI and intelligent automation, leveraging technologies including MLflow, Azure AI Foundry, Agent Bricks and Microsoft Copilot
Requirements for the Principal Data Engineer (Databricks)
- Extensive hands-on Azure Databricks expertise, with a strong track record designing and delivering enterprise-scale data platforms (PySpark, Python, SQL, Delta Lake, Structured Streaming, DLT)
- Proven ownership of data platform architecture, with strong knowledge of data modelling, integration, governance, security, performance and cost optimisation
- Experience leading complex, greenfield or large-scale data transformation programmes across multiple workstreams and engineering teams
- Strong real-time data and integration experience, including streaming architectures and complex data flows (Kafka beneficial)
- Excellent technical stakeholder management, with the ability to influence
Overview
High-profile role at the heart of a global trading data ecosystem.
You'll combine hands-on engineering with genuine technical influence, helping shape a highly secure, cloud-native data platform that connects trading desks, physical assets and external partners.
Contract Details
Initial 6-month contract with long-term programme scope , £850-950/day (inside IR35), hybrid in London.
Interviews are being scheduled immediately, with a view to onboarding as soon as possible, so please apply now for immediate consideration.