Lead Data Platform Engineer - Outside IR35

The Role

The core requirement is strong Lakehouse expertise, particularly Spark, PySpark, and SparkSQL. Hands-on experience in low-code is essential.

Data & Analytics Expertise

• Particularly relevant to your current Data Platform background:
• Data architecture (Lakehouse, Medallion, Kimball, 3NF).
• Data ingestion and processing.
• Data quality frameworks (Great Expectations, Soda).
• Data, BI and analytics platforms:

• Microsoft Fabric
• Purview
• APIs
• Streams

• Data governance and data literacy.
• KPI and semantic model design.
• AI and innovation in analytics.

Software Engineering: Strong knowledge of:

• Design, Build, Test, Release, Deploy using Azure Fabric, data engineering tools.
• Secure coding practices.
• API design and integration patterns.
• Integrations using APIs, ADF for managing wide variety of data from structured, unstructured to semi.

DevSecOps, Cloud & Infrastructure

Cloud platforms: Azure, AWS

CI/CD pipelines and automated deployments.

Infrastructure as Code (IaC):

• Terraform
• Git actions

• Containerisation:

• Docker

• Security practices:

• Identity and access management
• Secrets management

• Monitoring, logging, SLAs/SLOs.

Agile Delivery: Strong experience in:

• Agile, Scrum, and Lean methodologies.
• Backlog management and roadmap planning.
• Technical cross team dependency management.
• Continuous improvement of delivery performance.

Development Tools: Hands-on capability with:

• Pyspark, pytest, DAX, python, Azure functions, Lamda, and SQL.
• Git source control.
• Testing frameworks like great expectation, TDD, meta data driven.
• Build and dependency management tools.
• CI platforms.
• Terraform/Ansible (or similar).

Platform-Specific Expertise (Nice to have): The role expects deep expertise in at least one of:

• APIs & Integrations
• DevSecOps & Cloud Platforms
• Application Development (React, JavaScript/ECMAScript, PWA)
• QA Automation (Playwright, Selenium)
• Data Engineering
• Reporting & Analytics

Technical Leadership & Architecture (Nice to have)

Lead technical direction for products, platforms, and services.

Define and promote software development standards, patterns, and governance.

Drive architectural consistency and manage technical debt.

Review, establish and improve product software development life cycles.