Lead Data Platform Engineer - Outside IR35

The Role

The core requirement is strong Lakehouse expertise, particularly Spark, PySpark, and SparkSQL. A low-code candidate was previously put forward and rejected, so hands-on experience in these areas is essential.

Data & Analytics Expertise

• Particularly relevant to your current Data Platform background:
• Data architecture (Lakehouse, Medallion, Kimball, 3NF).
• Data ingestion and processing.
• Data quality frameworks (Great Expectations, Soda).
• Data, BI and analytics platforms:

• Microsoft Fabric
• Purview
• APIs
• Streams

• Data governance and data literacy.
• KPI and semantic model design.
• AI and innovation in analytics.

Software Engineering: Strong knowledge of:

• Design, Build, Test, Release, Deploy using Azure Fabric, data engineering tools.
• Secure coding practices.
• API design and integration patterns.
• Integrations using APIs, ADF for managing wide variety of data from structured, unstructured to semi.

• Cloud platforms: Azure, AWS
• CI/CD pipelines and automated deployments.
• Infrastructure as Code (IaC):

• Terraform
• Git actions

• Containerisation:

• Docker

• Security practices:

• Identity and access management
• Secrets management

• Monitoring, logging, SLAs/SLOs.

Agile Delivery: Strong experience in:

• Agile, Scrum, and Lean methodologies.
• Backlog management and roadmap planning.
• Technical cross team dependency management.
• Continuous improvement of delivery performance.

Development Tools: Hands-on capability with:

• Pyspark, pytest, DAX, python, Azure functions, Lamda, and SQL.
• Git source control.
• Testing frameworks like great expectation, TDD, meta data driven.
• Build and dependency management tools.
• CI platforms.
• Terraform/Ansible (or similar).

Platform-Specific Expertise (Nice to have): The role expects deep expertise in at least one of:

• APIs & Integrations
• DevSecOps & Cloud Platforms
• Application Development (React, JavaScript/ECMAScript, PWA)
• QA Automation (Playwright, Selenium)
• Data Engineering
• Reporting & Analytics

• Lead technical direction for products, platforms, and services.
• Define and promote software development standards, patterns, and governance.
• Drive architectural consistency and manage technical debt.
• Review, establish and improve product software development life cycles.