We are looking for an experienced Data Engineer with strong hands-on expertise in Databricks and Apache Spark to design, develop, and maintain scalable data engineering solutions. The ideal candidate will have extensive experience in building and optimising data pipelines, processing large datasets, implementing data transformation workflows, and developing reliable data platforms using Spark and cloud-based technologies.
The candidate should have strong programming skills in Python/PySpark and SQL, experience with distributed data processing, and the ability to work with cross-functional teams to deliver high-quality, scalable, and efficient data solutions.
Key Responsibilities
Strong hands-on experience with Databricks and Apache Spark in production environments.
Excellent programming skills in Python and PySpark.
Strong knowledge of Spark SQL, DataFrames, distributed processing, and performance tuning.
- Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
- Build and optimise ETL/ELT workflows for processing large volumes of structured and unstructured data.
- Develop complex data transformations using PySpark, Spark SQL, and Python.
- Work with Databricks notebooks, jobs, Delta Lake, and Lakehouse architecture.
- Implement data ingestion, cleansing, validation, transformation, and integration processes.
- Optimise Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.