Databricks Consultant

Summary: The role of Databricks Consultant involves leveraging expertise in Energy & Commodities Trading to design and optimize a trading data platform using Databricks Lakehouse. The consultant will act as a liaison between trading desks, quants, and engineering teams to deliver scalable, real-time data pipelines that support trading decisions and risk management. The position requires a strong background in energy trading and hands-on experience with Databricks technologies. The engagement is outside IR35 and based in London with a hybrid working arrangement.

Key Responsibilities:

• Build and optimize ETL/ELT pipelines using Databricks, Spark (PySpark/Scala), and Delta Lake for structured/unstructured energy market data.
• Partner with Quants, Risk Managers, and Traders to implement models, curve generation, position tracking, and risk metrics (eg, VaR).
• Tune clusters and SQL warehouses for low-latency, time-sensitive trading environments.
• Enforce governance, access controls, and compliance with UK/EU financial and energy regulations via Unity Catalog.
• Provide architectural best practices, mentor engineers, and advise stakeholders on maximizing Databricks ROI.

Key Skills:

• Proven background in energy trading firms, banks, utilities, or oil/gas majors with strong knowledge of physical/financial markets and trading data types.
• Hands-on experience with production workloads on Databricks (Azure/AWS), PySpark, Spark SQL, Delta Lake, Unity Catalog, Python, SQL, orchestration tools, and streaming.
• Strong consultative mindset, excellent communication, and ability to interface with traders and senior stakeholders.

Salary (Rate): undetermined

City: London

Country: UK

Working Arrangements: hybrid

IR35 Status: outside IR35

Seniority Level: undetermined

Industry: IT

Detailed Description From Employer:

Hiring! Databricks Consultant

• Role Title: Databricks Consultant
• Location: London, Hybrid
• Duration: Initial 3 months
• Rate: Market Related
• Engagement: Outside IR35

We are seeking a seasoned Databricks Consultant with deep expertise in Energy & Commodities Trading to design, accelerate, and optimize our trading data platform. Acting as the bridge between trading desks, quants, and engineering teams, you will leverage the Databricks Lakehouse to deliver scalable, Real Time pipelines powering trading decisions, risk management, and market analytics.

Key Responsibilities

Data Architecture: Build and optimize ETL/ELT pipelines using Databricks, Spark (PySpark/Scala), and Delta Lake for structured/unstructured energy market data.
Trading Analytics: Partner with Quants, Risk Managers, and Traders to implement models, curve generation, position tracking, and risk metrics (eg, VaR).
Performance Optimization: Tune clusters and SQL warehouses for low-latency, time-sensitive trading environments.
Data Governance: Enforce governance, access controls, and compliance with UK/EU financial and energy regulations via Unity Catalog.
Consultative Leadership: Provide architectural best practices, mentor engineers, and advise stakeholders on maximizing Databricks ROI.

Required Skills & Experience

• Energy Trading Expertise: Proven background in energy trading firms, banks, utilities, or oil/gas majors. Strong knowledge of physical/financial markets (Power, Gas, LNG, Crude, Carbon/Renewables) and trading data types (Reuters, Bloomberg, Trayport, weather, ETRM).
• Databricks Mastery: Hands-on experience with production workloads on Databricks (Azure/AWS), PySpark, Spark SQL, Delta Lake, Unity Catalog, Python, SQL, orchestration tools (Airflow, Workflows, ADF), and streaming (Kafka/Event Hubs).
• Professional Skills: Strong consultative mindset, excellent communication, and ability to interface with traders and senior stakeholders.