Senior Databricks Architect/Databricks Engineer- Contract- Remote
Posted Today by Jobserve
£700 Per day
Outside
Remote
United Kingdom
Summary: The role of Senior Databricks Architect/Databricks Engineer involves hands-on data engineering within the Databricks and Shared IEP Services data layer. The contractor will be responsible for delivering production-grade data pipelines and schemas, focusing on product data ingestion, AI feature enablement, and IEP & Medicaid integration. This is a long-term contract position with a minimum duration of 12 months, offering a competitive daily rate. The position is fully remote, allowing for flexibility in work arrangements.
Key Responsibilities:
- Build and maintain daily data pipelines for product data ingestion into the shared back-office data layer.
- Design and maintain schemas and data pipelines for AI feature enablement, including handling PII-sensitive data.
- Build and maintain pipelines for IEP, Medicaid, and related service data integration.
- Design and maintain Bronze, Silver, and Gold schemas and pipelines using Databricks Delta Lake.
- Orchestrate and support complex production pipelines with robust error handling and reconciliation.
- Implement fine-grained access control and data lineage using Unity Catalog governance.
- Ensure data cleanliness, integrity, and normalization across disparate source systems.
- Manage data privacy and compliance, including anonymizing and masking PII.
- Optimize performance and cost through techniques like Z-ordering and data skipping.
- Leverage senior engineering experience to guide data engineering practices.
Key Skills:
- Expertise in Medallion architecture and Databricks Delta Lake.
- Experience in production pipeline engineering and operational support.
- Knowledge of Unity Catalog governance and data security measures.
- Strong understanding of ETL challenges and cross-system data integration.
- Familiarity with data privacy regulations such as FERPA, COPPA, or GDPR.
- Ability to optimize performance and manage cloud costs effectively.
- 8+ years of software engineering experience, with 3+ years in a senior data engineering role.
- Familiarity with Databricks AI development capabilities is desirable.
- EdTech engineering experience and knowledge of relevant specifications and standards is a plus.
Salary (Rate): £700/day
City: undetermined
Country: United Kingdom
Working Arrangements: remote
IR35 Status: outside IR35
Seniority Level: Senior
Industry: IT
Detailed Description From Employer:
Senior Databricks Architect/Databricks Engineer- Contract- Remote
Rate- £700/day Outside ir35
Work from Home
Duration- 12 months minimum (with possible extensions)
This is a Senior Databricks hands on Architect who is a Databrick engineer by trade. This is a long-term contract engagementto provide dedicated data engineering depth within the Databricks and Shared IEP Services data layer, owning delivery across these workstreams and ensuring the resulting pipelines and schemas are production-grade, governed, and maintainable by the permanent team. The contractor will operate primarily within the Databricks and Shared IEP Services data layer, owning data engineering delivery across three areas:
1) Product data ingestion - Building and maintaining daily pipelines from content and assessment products into the shared back-office data layer.
2) AI feature enablement - Designing and maintaining the schemas and data pipelines that power LLM-based features (GenAI Goals, Bulk Upload, Evaluation Document processing), including handling of parsed, normalized, and PII-sensitive document data.
3) IEP & Medicaid integration - Building and maintaining pipelines for IEP, Medicaid, and related service data into Databricks and other IEP systems.
Essential Criteria
1. Medallion architecture depth - designing and maintaining Bronze, Silver, and Gold schemas and pipelines using Databricks Delta Lake, including schema evolution and enforcement.
2. Production pipeline engineering - orchestrating and operationally supporting complex pipelines; idempotency, robust error handling, and reconciliation.
3. Unity Catalog governance - fine-grained access control, data lineage, and row/column-level security.
4. Cross-system data integration - handling data cleanliness, integrity, normalization, backfilling, reconciliation, and other ETL challenges across disparate source systems
5. Data privacy & compliance - anonymizing, masking, or tokenizing PII; working understanding of FERPA, COPPA, or GDPR.
6. Performance & cost optimization - Z-ordering, liquid clustering, data skipping, and shuffle-partition management to control cloud spend.
7. Senior engineering experience - 8+ years overall software engineering, including 3+ years in a senior data engineering role.
Desirable Criteria
1. Familiarity with Databricks AI development capabilities (eg vector search, model serving, LLM pipelines).
2. EdTech engineering experience.
3. Familiarity with EdTech specifications and standards such as 1EdTech, EdFi, CEDS, or OneRoster.