Senior Databricks Architect/Lead Databricks Engineer
Posted Today by Jobserve
£700 Per day
Outside
Remote
London
Summary: The Senior Databricks Architect/Lead Databricks Engineer role is focused on delivering enterprise-scale data solutions within a modern Databricks ecosystem. This position requires a hands-on approach to data engineering, ensuring the development of robust, scalable, and production-grade data solutions while driving engineering excellence. The role involves collaboration with various teams and mentoring internal engineering teams to establish standards. It is part of a major long-term transformation programme with potential for extension.
Key Responsibilities:
- Architect, build and optimise enterprise-grade data solutions within Databricks
- Design and implement scalable Delta Lake data architectures following Medallion principles
- Lead data engineering delivery across multiple strategic workstreams
- Develop highly resilient, production-ready pipelines with strong operational controls
- Drive data governance, security, lineage and compliance best practices
- Collaborate with product, engineering and business teams to deliver trusted data assets
- Mentor internal engineering teams and establish engineering standards
- Design and maintain large-scale ingestion pipelines
- Integrate assessment, content and product data into shared enterprise data platforms
- Ensure data quality, consistency and operational reliability
- Build and maintain data foundations powering LLM and GenAI capabilities
- Design schemas and pipelines supporting AI-driven products and document processing
- Handle complex parsed, normalised and sensitive datasets at scale
- Support emerging AI use cases including evaluation processing, bulk uploads and intelligent workflows
- Build robust integrations across IEP, Medicaid and associated service platforms
- Deliver reconciliation, data quality and cross-system consistency
- Support enterprise reporting and operational analytics requirements
Key Skills:
- Deep expertise in Databricks and Delta Lake architecture
- Proven experience implementing and operating Medallion Architecture (Bronze, Silver, Gold)
- Advanced pipeline engineering experience including orchestration, idempotent processing, error handling, monitoring, recovery strategies, and data reconciliation
- Strong Unity Catalog experience including data governance, lineage, fine-grained permissions, row-level and column-level security
- Extensive ETL and data integration expertise across complex enterprise systems
- Strong understanding of data quality frameworks, data normalisation, backfilling strategies, and data integrity controls
- Experience handling sensitive and regulated data environments
- Knowledge of PII protection techniques including masking, tokenisation, and anonymisation
- Familiarity with GDPR and broader data privacy standards
- Performance and cost optimisation expertise including Z-Ordering, Liquid Clustering, Data Skipping, Query tuning, Shuffle optimisation, and Cloud cost control
- 8+ years software engineering experience
- 3+ years operating at Senior/Lead Data Engineer or Databricks Architect level
Salary (Rate): £700/day
City: London
Country: UK
Working Arrangements: remote
IR35 Status: outside IR35
Seniority Level: Senior
Industry: IT
Senior Databricks Architect/Lead Databricks Engineer
Remote (UK)
£700/day Outside IR35
Initial 12-Month Contract | Long-Term Programme | Poosible Extension
We're seeking an exceptional Senior Databricks Architect with a strong hands-on Data Engineering background to join a major long-term transformation programme.
This is not a pure architecture role. You'll be a Databricks specialist who remains close to the technology, designing and building enterprise-scale data solutions while driving engineering excellence across critical data platforms.
You'll take ownership of delivering robust, scalable, and production-grade data solutions within a modern Databricks ecosystem, ensuring platforms are governed, secure, maintainable, and optimised for long-term growth.
?? Key Responsibilities
Architect, build and optimise enterprise-grade data solutions within Databricks
Design and implement scalable Delta Lake data architectures following Medallion principles
Lead data engineering delivery across multiple strategic workstreams
Develop highly resilient, production-ready pipelines with strong operational controls
Drive data governance, security, lineage and compliance best practices
Collaborate with product, engineering and business teams to deliver trusted data assets
Mentor internal engineering teams and establish engineering standards
?? Core Workstreams
?? Product Data Ingestion
Design and maintain large-scale ingestion pipelines
Integrate assessment, content and product data into shared enterprise data platforms
Ensure data quality, consistency and operational reliability
?? AI & GenAI Feature Enablement
Build and maintain data foundations powering LLM and GenAI capabilities
Design schemas and pipelines supporting AI-driven products and document processing
Handle complex parsed, normalised and sensitive datasets at scale
Support emerging AI use cases including evaluation processing, bulk uploads and intelligent workflows
?? IEP & Medicaid Data Integration
Build robust integrations across IEP, Medicaid and associated service platforms
Deliver reconciliation, data quality and cross-system consistency
Support enterprise reporting and operational analytics requirements
?? Essential Experience
? Deep expertise in Databricks and Delta Lake architecture
? Proven experience implementing and operating Medallion Architecture (Bronze, Silver, Gold)
? Advanced pipeline engineering experience including:
Orchestration
Idempotent processing
Error handling
Monitoring
Recovery strategies
Data reconciliation
? Strong Unity Catalog experience including:
Data governance
Lineage
Fine-grained permissions
Row-level and column-level security
? Extensive ETL and data integration expertise across complex enterprise systems
? Strong understanding of:
Data quality frameworks
Data normalisation
Backfilling strategies
Data integrity controls
? Experience handling sensitive and regulated data environments
? Knowledge of PII protection techniques including:
Masking
Tokenisation
Anonymisation
? Familiarity with GDPR and broader data privacy standards
? Performance and cost optimisation expertise including:
Z-Ordering
Liquid Clustering
Data Skipping
Query tuning
Shuffle optimisation
Cloud cost control
? 8+ years software engineering experience
? 3+ years operating at Senior/Lead Data Engineer or Databricks Architect level
?? Highly Desirable
? Databricks AI capabilities
Vector Search
Model Serving
LLM Pipelines
Retrieval-Augmented Generation (RAG)
? EdTech domain experience
? Experience with industry standards such as:
Ed-Fi
OneRoster
CEDS
1EdTech
High-impact programme combining Data Engineering, AI and Enterprise Architecture
Opportunity to shape the future data platform of a growing organisation
Significant ownership and influence from day one