Senior Data Engineer & Technical Delivery Lead (Python/Databricks)

Key Responsibilities

• Design, develop and maintain scalable data pipelines using Python and Databricks.
• Build, optimise and support enterprise ETL/ELT workflows using Apache Spark and Delta Lake.
• Design and develop robust data models and Lakehouse architectures.
• Implement and manage data workflows within the Databricks ecosystem.
• Ensure high levels of data quality, governance and pipeline reliability.
• Optimise performance across large-scale distributed data processing platforms.
• Collaborate with architects, product owners, analysts and engineering teams to deliver high-quality data solutions.
• Implement monitoring, logging and alerting across critical data platforms.
• Drive engineering best practice through code reviews, mentoring and technical collaboration.
• For Technical Delivery Lead opportunities, provide technical leadership, architectural oversight and end-to-end delivery of complex data engineering initiatives.

Required Skills & Experience

• Strong commercial experience developing data engineering solutions using Python.
• Extensive hands-on experience with Databricks, including Workflows, Notebooks and Delta Lake.
• Strong experience with Apache Spark/PySpark.
• Proven experience building scalable ETL/ELT pipelines.
• Strong SQL and data modelling skills.
• Experience designing modern Lakehouse architectures.
• Experience working with cloud platforms including AWS, Azure or GCP.
• Experience with modern data warehousing technologies such as Snowflake, Redshift or BigQuery.
• Experience with orchestration tools such as Airflow or Databricks Workflows.
• Previous experience working within Agile delivery environments.

Desirable Experience

• Databricks certifications.
• Kafka or Structured Streaming.
• CI/CD and DevOps practices.
• Docker and Kubernetes.
• Exposure to Machine Learning pipelines.
• Experience within Financial Services or other highly regulated environments.

About You

You'll be an experienced Data Engineer or Technical Lead with a passion for building modern, scalable data platforms.

Comfortable working in a collaborative Agile environment, you'll enjoy solving complex technical problems and delivering high-quality engineering solutions.

If you're applying at Technical Delivery Lead level, you'll also bring strong leadership, stakeholder management and mentoring experience whilst remaining hands-on technically.