Senior Data Engineer - PySpark - Airflow
We're looking for an experienced Senior Data Engineer to join a specialist financial services business on an urgent 6-month contract.
You'll be working within the Actuarial Analytics team, helping to develop and improve a modern analytics data platform. The role will involve building scalable data pipelines, transforming complex datasets and delivering high-quality data products that support business-critical reporting and analysis.
This is a hands-on role where you'll work closely with technical teams and business stakeholders, taking ownership of data engineering projects from requirements through to delivery.
Key responsibilities
- Design, build and maintain scalable, secure data pipelines and data products.
- Develop and optimise data transformations using Python, PySpark and DBT.
- Work with Lakehouse architectures and Medallion frameworks to deliver clean, analysis-ready data.
- Implement automated data quality checks, monitoring and robust DataOps processes.
- Work closely with stakeholders to understand requirements and translate business needs into technical solutions.
- Enhance and maintain existing data products while identifying opportunities to improve performance, efficiency and scalability.
- Apply best practices across data modelling, warehousing, documentation and engineering standards.
Key skills and experience
- Strong commercial experience as a Data Engineer or in a similar role.
- Hands-on experience with Airflow.
- Strong Python skills, ideally including PySpark.
- Proven experience building and maintaining data pipelines and data products.
- Good understanding of data modelling, data warehousing and data quality.
- Experience with data transformation, orchestration and modern data platform architectures.
- Strong communication skills and the ability to work directly with business stakeholders.
Desirable experience
- Microsoft Azure data technologies, such as Azure Data Factory, Azure Data Lake Storage, Fabric or Synapse Analytics.
- Lakehouse architecture and the Medallion framework (Bronze, Silver and Gold).
- Airflow or similar orchestration tools.
- Experience working with actuarial, insurance or other financial services data.
Contract details
- Rate: £750 per day
- IR35: Inside IR35
- Duration: Initial 6 months
- Location: London
- Start date: Immediate
This is an urgent requirement, so candidates who are available to start immediately or at short notice are strongly preferred.
If you have strong DBT and Python experience and are available for your next contract, please get in touch to discuss the role.