Senior Databricks Architect/Databricks Engineer- Contract- Remote
Posted 3 days ago by Careerwise
£700 Per day
Outside
Remote
London, UK
Summary: The role of Senior Databricks Architect/Engineer involves providing hands-on expertise in data engineering within the Databricks and Shared IEP Services data layer. The contractor will be responsible for building and maintaining data pipelines, ensuring production-grade quality, and integrating various data sources. This long-term contract position requires a deep understanding of medallion architecture and data privacy compliance. The role is fully remote and offers a competitive daily rate.
Key Responsibilities:
- Build and maintain daily data ingestion pipelines from content and assessment products.
- Design and maintain schemas and data pipelines for AI feature enablement.
- Integrate IEP, Medicaid, and related service data into Databricks and other systems.
- Design and maintain Bronze, Silver, and Gold schemas and pipelines using Databricks Delta Lake.
- Orchestrate and support complex production pipelines with robust error handling.
- Implement Unity Catalog governance for data access control and lineage.
- Handle data integration challenges across disparate source systems.
- Ensure data privacy and compliance with regulations like FERPA, COPPA, or GDPR.
- Optimize performance and cost management in cloud environments.
- Leverage senior engineering experience in software and data engineering roles.
Key Skills:
- Expertise in medallion architecture and Databricks Delta Lake.
- Experience in production pipeline engineering and operational support.
- Knowledge of Unity Catalog governance and data security.
- Proficiency in cross-system data integration and ETL processes.
- Understanding of data privacy and compliance regulations.
- Skills in performance and cost optimization techniques.
- 8+ years of software engineering experience, with 3+ years in a senior data engineering role.
- Familiarity with Databricks AI development capabilities (desirable).
- EdTech engineering experience (desirable).
- Knowledge of EdTech specifications and standards (desirable).
Salary (Rate): £700/day
City: London
Country: UK
Working Arrangements: remote
IR35 Status: outside IR35
Seniority Level: Senior
Industry: IT
Senior Databricks Architect/Databricks Engineer- Contract- Remote
Rate- £700/day Outside IR35
Work from Home
Duration- 12 months minimum (with possible extensions)
This is a Senior Databricks hands-on Architect who is a Databricks engineer by trade. This is a long-term contract engagementto provide dedicated data engineering depth within the Databricks and Shared IEP Services data layer, owning delivery across these workstreams and ensuring the resulting pipelines and schemas are production-grade, governed, and maintainable by the permanent team. The contractor will operate primarily within the Databricks and Shared IEP Services data layer, owning data engineering delivery across three areas:
1) Product data ingestion - Building and maintaining daily pipelines from content and assessment products into the shared back-office data layer.
2) AI feature enablement - Designing and maintaining the schemas and data pipelines that power LLM-based features (GenAI Goals, Bulk Upload, Evaluation Document processing), including handling of parsed, normalized, and PII-sensitive document data.
3) IEP & Medicaid integration - Building and maintaining pipelines for IEP, Medicaid, and related service data into Databricks and other IEP systems.
Essential Criteria
1. Medallion architecture depth - designing and maintaining Bronze, Silver, and Gold schemas and pipelines using Databricks Delta Lake, including schema evolution and enforcement.
2. Production pipeline engineering - orchestrating and operationally supporting complex pipelines; idempotency, robust error handling, and reconciliation.
3. Unity Catalog governance - fine-grained access control, data lineage, and row/column-level security.
4. Cross-system data integration - handling data cleanliness, integrity, normalization, backfilling, reconciliation, and other ETL challenges across disparate source systems
5. Data privacy & compliance - anonymizing, masking, or tokenizing PII; working understanding of FERPA, COPPA, or GDPR.
6. Performance & cost optimization - Z-ordering, liquid clustering, data skipping, and shuffle-partition management to control cloud spend.
7. Senior engineering experience - 8+ years overall software engineering, including 3+ years in a senior data engineering role.
Desirable Criteria
1. Familiarity with Databricks AI development capabilities (eg vector search, model serving, LLM pipelines).
2. EdTech engineering experience.
3. Familiarity with EdTech specifications and standards such as 1EdTech, EdFi, CEDS, or OneRoster.