Data Engineer – Azure Databricks / Python / Insurance - Outside IR35
Posted Today by Totaljobs
Negotiable
Outside
Remote
UK; London; Birmingham; Manchester; Glasgow; Liverpool; Leeds; Edinburgh; Bristol; Cardiff; Nottingham
Summary: The Data Engineer role focuses on designing and implementing a data lake architecture using Azure and Databricks for a leading client in the financial services sector, specifically insurance. The position is fully remote and is expected to last 2-3 months with the possibility of extension. The engineer will be responsible for developing ingestion patterns, establishing data governance, and building data transformation logic. This role requires experience in Azure, Databricks, and Python, particularly within the financial services industry.
Key Responsibilities:
- Define and build a data lake architecture on Azure (ADLS + Databricks)
- Implement layered data model (bronze / silver / gold) for ingestion, processing, and reporting
- Develop ingestion patterns from API sources (batch and streaming)
- Establish data governance, metadata, and access controls
- Define storage structures, naming conventions, and schema design
- Develop API connectors and ingestion pipelines using Python
- Implement integration services feeding into the Databricks environment
- Build data transformation logic using PySpark / Python (data cleansing, standardisation, validation)
- Configure monitoring, logging, and alerting
- Execute unit and integration testing across pipelines
Key Skills:
- Azure
- Databricks
- Python
- Financial Services sector experience – Ideally Insurance / General or Life Insurance
Salary (Rate): undetermined
City: undetermined
Country: UK
Working Arrangements: remote
IR35 Status: outside IR35
Seniority Level: undetermined
Industry: IT
Data Engineer / Data Engineering / Azure / Databricks / Ingestion / Pipelines / Insurance / Financial Services / 2-3 month contract / Outside IR35 / Fully remote
One of our leading clients is looking to recruit a Data Engineer.
Location – fully remote / home based
Duration – 2-3 months plus possible extension
Outside IR35
Role / Responsibilities:
- Define and build a data lake architecture on Azure (ADLS + Databricks)
- Implement layered data model (bronze / silver / gold) for ingestion, processing, and reporting
- Develop ingestion patterns from API sources (batch and streaming)
- Establish data governance, metadata, and access controls
- Define storage structures, naming conventions, and schema design
- Develop API connectors and ingestion pipelines using Python
- Implement integration services feeding into the Databricks environment
- Build data transformation logic using PySpark / Python (data cleansing, standardisation, validation)
- Configure monitoring, logging, and alerting
- Execute unit and integration testing across pipelines
Experience:
- Azure
- Databricks
- Python
- Financial Services sector experience – Ideally Insurance / General or Life Insurance