The Role
Depending on your level, you'll take ownership of designing, building and/or leading modern Databricks data platforms, working across large-scale data engineering, platform architecture, governance and deployment.
You'll work with engineering, architecture and business teams to deliver scalable production-grade solutions while helping establish strong engineering standards across the wider team.
Key Responsibilities
- Design and build production-grade data pipelines using Databricks, PySpark and Spark SQL
- Design and implement Delta Lake/Medallion Architecture solutions
- Lead technical design across ingestion, transformation and data consumption
- Implement and optimise Unity Catalog for governance, lineage and access control
- Build and maintain CI/CD and Infrastructure-as-Code using Terraform/Databricks Asset Bundles
- Optimise Spark workloads for performance, scalability and cost
- Establish engineering standards around testing, deployment and code quality
- Work closely with architects, engineers and client stakeholders
- Mentor and technically support other engineers
- At Principal level, own platform-level architecture and set technical direction across Databricks workstreams
Essential Experience: We're particularly interested in contractors with:
- Strong commercial Databricks experience
- Advanced PySpark and Spark SQL
- Strong understanding of Spark internals and performance optimisation
- Hands-on Delta Lake and Medallion Architecture experience
- Strong Python engineering skills
- Hands-on Unity Catalog experience
- CI/CD and Infrastructure-as-Code experience
- Terraform and/or Databricks Asset Bundles
- Cloud experience across Azure, AWS or GCP
- Experience delivering large-scale production data platforms
- Strong stakeholder communication skills
For the Principal-level positions
you should also have experience setting technical direction, owning platform architecture and operating as the technical authority across large Databricks engagements.
Desirable
- Databricks Professional-level certifications
- Delta Sharing/Lakehouse Federation
- Structured Streaming/Kafka
- MLflow or AI/ML-adjacent data engineering
- Agentic or AI-driven data engineering patterns
- Previous consultancy or contract delivery experience
Contract Details
Senior Data Engineer: £550-£625 per day
Principal Data Engineer: Up to £700 per day
IR35: Outside IR35
Location: Fully Remote - UK
Start: ASAP
Contract: 3-6 Initial with potential for extension