Databricks Pipeline Engineer
Posted 1 day ago by Access Computer Consulting Plc
£450 Per day
Inside
Remote
City of London, UK
Summary: The Databricks Pipeline Engineer role is a remote position focused on designing, optimizing, and maintaining scalable Databricks pipelines, including ETL, streaming, and ML workflows. The position requires strong expertise in Databricks, performance tuning, and cost-optimization strategies. Familiarity with orchestration tools and proficiency in Python or Scala is essential. The role falls inside IR35, necessitating work through an umbrella company.
Key Responsibilities:
- Design, optimize, and maintain scalable Databricks pipelines (ETL, streaming, ML workflows).
- Perform cluster and job performance tuning, including optimizing cluster sizing, caching, partitioning, and shuffle management.
- Monitor Spark job metrics, analyze logs, and identify bottlenecks in data throughput or latency.
- Implement cost-optimization strategies for Databricks jobs and clusters using autoscaling and job consolidation.
- Utilize orchestration tools such as Databricks Workflows, Airflow, or Azure Data Factory.
- Develop data engineering and pipeline solutions using Python or Scala.
- Manage hands-on experience with Azure, AWS, or multi-cloud Databricks deployments.
- Understand data storage layers (Azure Data Lake Storage, AWS S3) and performance trade-offs.
- Utilize version control (Git, GitHub Actions, DevOps pipelines) and CI/CD practices for Databricks.
Key Skills:
- Strong expertise in Databricks Pipelines.
- Experience in designing, optimizing, and maintaining scalable Databricks pipelines.
- Proficiency in Python or Scala for data engineering and pipeline development.
- Familiarity with orchestration tools: Databricks Workflows, Airflow, or Azure Data Factory.
- Hands-on experience with Azure, AWS, or multi-cloud Databricks deployments.
- Knowledge of data storage layers (Azure Data Lake Storage, AWS S3) and performance trade-offs.
- Experience with version control (Git, GitHub Actions, DevOps pipelines) and CI/CD practices.
Salary (Rate): £450 per day
City: City of London
Country: UK
Working Arrangements: remote
IR35 Status: inside IR35
Seniority Level: undetermined
Industry: IT
I am recruiting for a Databricks Pipeline Engineer to work on a remote basis.
The role falls inside IR35 so you will be required to work through an umbrella company for the duration of the contract.
Strong expertise in Databricks Pipelines.
You must have experience of designing, optimising and maintaining scalable Databricks pipelines (ETL, streaming, ML workflows).
You will be able to perform cluster and job performance tuning: optimise cluster sizing, caching, partitioning, and shuffle management.
You will monitor Spark job metrics, analyse logs, and identify bottlenecks in data throughput or latency.
You will also be able to implement cost-optimisation strategies for Databricks jobs and clusters using autoscaling and job consolidation.
You must have familiarity with orchestration tools: Databricks Workflows, Airflow, or Azure Data Factory.
Proficiency in Python or Scala for data engineering and pipeline development is essential.
Hands-on experience with Azure, AWS, or multi-cloud Databricks deployments.
Knowledge of data storage layers (Azure Data Lake Storage, AWS S3) and performance trade-offs.
Experience of version control (Git, GitHub Actions, DevOps pipelines) and CI/CD practices for Databricks.
Please apply ASAP to find out more.
