Databricks Consultant (UC Migration)

Databricks Consultant (UC Migration)

Posted Today by Hays

Apply
Negotiable
Outside
Hybrid
Swindon
Apply
Amazon Web Services (AWS) Apache Hive Azure Databricks Databricks Data Store Grafana Infrastructure as Code (IaC) Kibana Nice (Unix Utility) Python (Programming Language) SQL (Programming Language) Stakeholder Engagement Stakeholder Management Terraform

Summary: Hays is seeking a highly experienced Databricks Consultant/SME for a local client, focusing on leading migrations from legacy setups to Unity Catalog. The role involves optimizing the Databricks environment and automating infrastructure using Terraform or CloudFormation. The position is primarily remote with one day a week in Swindon and is outside IR35. An initial contract of six months is offered, with potential for extensions.

Key Responsibilities:

  • Lead the migration of Databricks assets (notebooks, jobs, clusters) from DBFS to Unity Catalog
  • Define and implement a scalable, standardised migration approach across teams
  • Support teams with ad-hoc Databricks issues related to migration and platform usage
  • Optimise the Databricks environment for structure, access, and governance
  • Automate infrastructure and deployments using Terraform and/or CloudFormation
  • Build and maintain data pipelines using Python, SQL and Databricks APIs
  • Monitor platform performance using tools such as CloudWatch, Grafana, and Kibana
  • Collaborate with engineering teams, stakeholders, and platform users

Key Skills:

  • Strong hands-on experience with Databricks in an AWS environment
  • Proven experience with Unity Catalog and Databricks migrations
  • Strong Python skills and experience working with Databricks APIs
  • Solid understanding of data pipelines, SQL, and data platform architecture
  • Experience with Infrastructure as Code (Terraform or CloudFormation)
  • Knowledge of monitoring and observability tools (e.g. CloudWatch, Grafana, Kibana)
  • Strong problem-solving skills and ability to work independently
  • Excellent communication and stakeholder engagement skills
  • Exposure to Databricks Apps or AI/ML workloads (nice to have)

Salary (Rate): undetermined

City: Swindon

Country: UK

Working Arrangements: hybrid

IR35 Status: outside IR35

Seniority Level: undetermined

Industry: IT

Detailed Description From Employer:

Hays are now looking for a Databricks Consultant / SME for one of our local clients:

Flexible rate - Outside IR35
1 day a week in Swindon, the rest is WFH
Initial 6 month contract (extensions likely)

We’re seeking a highly experienced Databricks specialist who can confidently navigate the platform end-to-end, including leading migrations from legacy setups using DBFS-mounted tables and Hive Metastore. You’ll bring strong AWS knowledge, hands-on experience working with S3 via Python, a solid understanding of data storage formats, and practical Python development expertise.

What you’ll be doing

  • Lead the migration of Databricks assets (notebooks, jobs, clusters) from DBFS to Unity Catalog
  • Define and implement a scalable, standardised migration approach across teams
  • Support teams with ad-hoc Databricks issues related to migration and platform usage
  • Optimise the Databricks environment for structure, access, and governance
  • Automate infrastructure and deployments using Terraform and/or CloudFormation
  • Build and maintain data pipelines using Python, SQL and Databricks APIs
  • Monitor platform performance using tools such as CloudWatch, Grafana, and Kibana
  • Collaborate with engineering teams, stakeholders, and platform users

What we’re looking for

  • Strong hands-on experience with Databricks in an AWS environment
  • Proven experience with Unity Catalog and Databricks migrations
  • Strong Python skills and experience working with Databricks APIs
  • Solid understanding of data pipelines, SQL, and data platform architecture
  • Experience with Infrastructure as Code (Terraform or CloudFormation)
  • Knowledge of monitoring and observability tools (e.g. CloudWatch, Grafana, Kibana)
  • Strong problem-solving skills and ability to work independently
  • Excellent communication and stakeholder engagement skills
  • Exposure to Databricks Apps or AI/ML workloads (nice to have)

If you have previously led migrations to Unity Catalogue and you're interested in this role, click 'apply now' to forward an up-to-date copy of your CV, or please send a copy of your CV to jonathan.davies1@hays.com

Apply
Similar Jobs
Loading data...