Senior Databricks Engineer
Posted 3 days ago by Lorien
£405 Per day
Inside
Undetermined
Glasgow, Scotland, UK
Summary: The Senior Databricks Engineer role involves leading the migration of data pipelines from AWS to the Databricks platform. This hands-on position focuses on designing, building, and optimizing scalable data solutions while ensuring data quality and governance. The engineer will collaborate with cross-functional teams and provide mentorship to junior engineers. The role requires strong expertise in Databricks and Apache Spark, along with experience in cloud environments.
Key Responsibilities:
- Lead the migration of existing AWS-based data pipelines to Databricks.
- Design and implement scalable data engineering solutions using Apache Spark on Databricks.
- Collaborate with cross-functional teams to understand data requirements and translate them into efficient pipelines.
- Optimize performance and cost-efficiency of Databricks workloads.
- Develop and maintain CI/CD workflows for Databricks using GitLab or similar tools.
- Ensure data quality and reliability through robust unit testing and validation frameworks.
- Implement best practices for data governance, security, and access control within Databricks.
- Provide technical mentorship and guidance to junior engineers.
Key Skills:
- Strong hands-on experience with Databricks and Apache Spark (preferably PySpark).
- Proven track record of building and optimizing data pipelines in cloud environments.
- Experience with AWS services such as S3, Glue, Lambda, Step Functions, Athena, IAM, and VPC.
- Proficiency in Python for data engineering tasks.
- Familiarity with GitLab for version control and CI/CD.
- Strong understanding of unit testing and data validation techniques.
Salary (Rate): £405 per day
City: Glasgow
Country: UK
Working Arrangements: undetermined
IR35 Status: inside IR35
Seniority Level: Senior
Industry: IT
Role Title: Sr.Databricks Engineer
Location: Glasgow
Days on site: 2-3
Pay Rate: £405 per day
MUST BE PAYE THROUGH UMBRELLA
Role Description:
We are currently migrating our data pipelines from AWS to Databricks, and are seeking a Senior Databricks Engineer to lead and contribute to this transformation. This is a hands-on engineering role focused on designing, building, and optimizing scalable data solutions using the Databricks platform.
Key Responsibilities:
. Lead the migration of existing AWS-based data pipelines to Databricks.
. Design and implement scalable data engineering solutions using Apache Spark on Databricks.
. Collaborate with cross-functional teams to understand data requirements and translate them into efficient pipelines.
. Optimize performance and cost-efficiency of Databricks workloads.
. Develop and maintain CI/CD workflows for Databricks using GitLab or similar tools.
. Ensure data quality and reliability through robust unit testing and validation frameworks.
. Implement best practices for data governance, security, and access control within Databricks.
. Provide technical mentorship and guidance to junior engineers.
Must-Have Skills:
. Strong hands-on experience with Databricks and Apache Spark (preferably PySpark).
. Proven track record of building and optimizing data pipelines in cloud environments.
. Experience with AWS services such as S3, Glue, Lambda, Step Functions, Athena, IAM, and VPC.
. Proficiency in Python for data engineering tasks.
. Familiarity with GitLab for version control and CI/CD.
. Strong understanding of unit testing and data validation techniques.
Preferred Qualifications:
. Experience with Databricks Delta Lake, Unity Catalog, and MLflow.
. Knowledge of CloudFormation or other infrastructure-as-code tools.
. AWS or Databricks certifications.
. Experience in large-scale data migration projects.
. Background in Finance Industry.
Carbon60, Lorien & SRG - The Impellam Group STEM Portfolio are acting as an Employment Business in relation to this vacancy.
