PySpark Developer (Microsoft Fabric) - 6-Month Contract - London
Posted Today by Hamilton Barnes
£450 Per day
Inside
Hybrid
London, UK
Summary: The role of a PySpark Developer involves designing, building, and maintaining data pipelines for a leading global financial markets infrastructure provider. This 6-month hybrid contract position requires expertise in PySpark, Azure Cosmos DB, and Microsoft Fabric. The developer will be responsible for ensuring data quality and optimizing data processes while translating business requirements into technical solutions. The position is based in London, with a flexible work arrangement of 1-3 days in the office.
Key Responsibilities:
- Build and maintain PySpark pipelines for ingestion, normalisation, bi-temporal processing, and publication of financial instrument data
- Develop and optimise Azure Cosmos DB read/write patterns and integrate with external APIs for Minting & Matching
- Implement data quality checks, comprehensive unit/integration tests, and CI/CD pipeline management via GitLab CI
- Monitor and troubleshoot production pipelines using Eventstream telemetry, KQL, and DataDog
- Translate business requirements into technical solutions, maintaining documentation and knowledge repositories
Key Skills:
- Strong proficiency in Python and PySpark, with hands-on experience building streaming mastering pipelines
- Solid Azure experience - Cosmos DB (NoSQL), ADLS/OneLake, and Key Vault
- Strong understanding of temporal data modelling, data quality frameworks, and CI/CD practices
- Experience with lakehouses, dataflows, and semantic models within Microsoft Fabric - preferred
- Domain knowledge of financial instruments and reference data - advantageous
Salary (Rate): £450 per day
City: London
Country: UK
Working Arrangements: hybrid
IR35 Status: inside IR35
Seniority Level: undetermined
Industry: IT
Detailed Description From Employer:
PySpark Developer (Microsoft Fabric) - 6-Month Contract - London
Contracting role based in London (Hybrid, 1-3 days in office) for a 6-month engagement. You will design, build, and maintain data pipelines powering an end-to-end mastering flow for a leading global financial markets infrastructure and data provider, working across PySpark, Azure Cosmos DB, and Microsoft Fabric.
Key Responsibilities
- Build and maintain PySpark pipelines for ingestion, normalisation, bi-temporal processing, and publication of financial instrument data
- Develop and optimise Azure Cosmos DB read/write patterns and integrate with external APIs for Minting & Matching
- Implement data quality checks, comprehensive unit/integration tests, and CI/CD pipeline management via GitLab CI
- Monitor and troubleshoot production pipelines using Eventstream telemetry, KQL, and DataDog
- Translate business requirements into technical solutions, maintaining documentation and knowledge repositories
What You Will Ideally Bring
- Strong proficiency in Python and PySpark, with hands-on experience building streaming mastering pipelines
- Solid Azure experience - Cosmos DB (NoSQL), ADLS/OneLake, and Key Vault
- Strong understanding of temporal data modelling, data quality frameworks, and CI/CD practices
- Experience with lakehouses, dataflows, and semantic models within Microsoft Fabric - preferred
- Domain knowledge of financial instruments and reference data - advantageous
Contract Details
- Duration: 6 months (with potential extension)
- Day Rate: Up to £450 per day (Inside IR35)
- Location: - Hybrid London
- Start Date: ASAP