Senior Databricks Architect/Databricks Engineer- Contract- Remote
Posted 2 weeks ago by Careerwise
£700 Per day
Outside
Remote
United Kingdom
Summary: The role of Senior Databricks Architect/Databricks Engineer involves hands-on data engineering within the Databricks and Shared IEP Services data layer, focusing on building and maintaining production-grade data pipelines. The contractor will be responsible for product data ingestion, AI feature enablement, and IEP & Medicaid integration, ensuring data governance and compliance. This is a long-term contract position with a minimum duration of 12 months, offering a competitive daily rate. The position is fully remote, allowing for flexible work arrangements.
Key Responsibilities:
- Build and maintain daily pipelines for product data ingestion into the shared Back Office data layer.
- Design and maintain schemas and data pipelines for AI feature enablement, including handling PII-sensitive data.
- Build and maintain pipelines for IEP, Medicaid, and related service data into Databricks and other IEP systems.
- Design and maintain Bronze, Silver, and Gold schemas and pipelines using Databricks Delta Lake.
- Orchestrate and operationally support complex production pipelines with robust error handling.
- Implement fine-grained access control and data lineage using Unity Catalog governance.
- Handle data cleanliness, integrity, normalization, and reconciliation across disparate source systems.
- Ensure data privacy and compliance with regulations such as FERPA, COPPA, or GDPR.
- Optimize performance and cost management of data processes.
- Leverage senior engineering experience to guide data engineering practices.
Key Skills:
- Expertise in Medallion architecture and Databricks Delta Lake.
- Strong experience in production pipeline engineering and operational support.
- Knowledge of Unity Catalog governance and data security practices.
- Experience with cross-system data integration and ETL challenges.
- Understanding of data privacy and compliance regulations.
- Skills in performance and cost optimization techniques.
- 8+ years of overall software engineering experience, with 3+ years in a senior data engineering role.
- Familiarity with Databricks AI development capabilities is desirable.
- EdTech engineering experience is a plus.
- Knowledge of EdTech specifications and standards such as 1EdTech, EdFi, CEDS, or OneRoster is beneficial.
Salary (Rate): £700/day
City: undetermined
Country: United Kingdom
Working Arrangements: remote
IR35 Status: outside IR35
Seniority Level: Senior
Industry: IT
Detailed Description From Employer:
Senior Databricks Architect/Databricks Engineer- Contract- Remote
Rate- £700/day Outside ir35
Work from Home
Duration- 12 months minimum (with possible extensions)
This is a Senior Databricks hands on Architect who is a Databrick engineer by trade. This is a long-term contract engagementto provide dedicated data engineering depth within the Databricks and Shared IEP Services data layer, owning delivery across these workstreams and ensuring the resulting pipelines and schemas are production-grade, governed, and maintainable by the permanent team. The contractor will operate primarily within the Databricks and Shared IEP Services data layer, owning data engineering delivery across three areas:
1) Product data ingestion - Building and maintaining daily pipelines from content and assessment products into the shared Back Office data layer.
2) AI feature enablement - Designing and maintaining the schemas and data pipelines that power LLM-based features (GenAI Goals, Bulk Upload, Evaluation Document processing), including handling of parsed, normalized, and PII-sensitive document data.
3) IEP & Medicaid integration - Building and maintaining pipelines for IEP, Medicaid, and related service data into Databricks and other IEP systems.
Essential Criteria
1. Medallion architecture depth - designing and maintaining Bronze, Silver, and Gold schemas and pipelines using Databricks Delta Lake, including schema evolution and enforcement.
2. Production pipeline engineering - orchestrating and operationally supporting complex pipelines; idempotency, robust error handling, and reconciliation.
3. Unity Catalog governance - fine-grained access control, data lineage, and row/column-level security.
4. Cross-system data integration - handling data cleanliness, integrity, normalization, backfilling, reconciliation, and other ETL challenges across disparate source systems
5. Data privacy & compliance - anonymizing, masking, or tokenizing PII; working understanding of FERPA, COPPA, or GDPR.
6. Performance & cost optimization - Z-ordering, liquid clustering, data skipping, and shuffle-partition management to control cloud spend.
7. Senior engineering experience - 8+ years overall software engineering, including 3+ years in a senior data engineering role.
Desirable Criteria
1. Familiarity with Databricks AI development capabilities (eg vector search, model serving, LLM pipelines).
2. EdTech engineering experience.
3. Familiarity with EdTech specifications and standards such as 1EdTech, EdFi, CEDS, or OneRoster.