Lead Data Engineer - Birmingham - 6 months - Hybrid

Lead Data Engineer - Birmingham - 6 months - Hybrid

Posted Today by Simon James I.T. Ltd

Apply
£450 Per day
Undetermined
Hybrid
Birmingham, UK
Apply
Apache Beam Apache Kafka Application Programming Interface (API) Automation AWS Identity And Access Management (IAM) AWS Key Management Service (KMS) BigQuery Change Data Capture (CDC) Cloud Computing Cloud Technology Continuous Integration and Continuous Delivery Dataflow Data Loss Prevention Data Quality Digital Transformation Google Cloud Google Cloud Platform (GCP) Google Kubernetes Engine (GKE) Java (Programming Language) Python (Programming Language) Real Time Data SQL (Programming Language) Terraform

Summary: The Lead Data Engineer role focuses on architecting and driving near real-time ingestion platforms on Google Cloud Platform (GCP) for a major UK telecom undergoing digital transformation. The position involves solving distributed systems challenges and leading engineering teams to deliver robust data systems at scale. Candidates should have a strong background in GCP technologies and experience in building low-latency streaming pipelines. This is a pivotal opportunity for those passionate about real-time data platforms.

Key Responsibilities:

  • Build and evolve near Real Time ingestion frameworks using Pub/Sub, Kafka, Dataflow (Beam), GCS, BigQuery, Cloud SQL.
  • Architect fast, reliable ingestion pipelines with Apache Beam (Java/Python) on Dataflow.
  • Deliver robust CDC ingestion using Datastream, handling schema drift and low-latency data updates.
  • Apply deep BigQuery optimisation: Storage Write API, partitioning, clustering, materialised views.
  • Optimise streaming workloads for latency, throughput, backpressure, watermarks, and windowing.
  • Build self-healing, high-availability pipelines with enterprise-grade SLAs.
  • Establish strong observability: Cloud Monitoring, Logging, Trace, DLQs, replay strategies, retry frameworks.
  • Implement secure architectures across IAM, KMS, VPC-SC, DLP, Secret Manager.
  • Champion governance, data quality, and auditability.
  • Lead the NRT roadmap, mentor engineers, and partner with architecture and analytics teams.
  • Drive engineering excellence, reusable frameworks, and automation.

Key Skills:

  • Deep hands-on expertise with GCP - Pub/Sub, Dataflow, BigQuery, GCS, Composer.
  • Strong background in CDC technologies (Datastream preferred).
  • Advanced Apache Beam engineering skills (Java or Python).
  • Solid grounding in Terraform, CI/CD, GitOps, and cloud-native orchestration (eg, GKE).
  • Proven leadership in designing and scaling complex streaming ingestion systems.

Salary (Rate): £450 daily

City: Birmingham

Country: UK

Working Arrangements: hybrid

IR35 Status: undetermined

Seniority Level: undetermined

Industry: IT

Detailed Description From Employer:

Lead Data Engineer (Near Real Time Ingestion, GCP)

Location: Birmingham (Hybrid - 2 days onsite)

Are you ready to build and lead near Real Time data platforms powering one of the UK's largest telecom infrastructures?

We're hiring for a major national telco undergoing a high-profile digital transformation. We're looking for a Lead Data Engineer to architect and drive their NRT ingestion platforms on GCP, working on cutting-edge, mission-critical data systems operating at massive scale.

If you love solving distributed systems challenges, designing low-latency streaming pipelines, and leading high-performing engineering teams, this is the role for you.

What You'll Own & Deliver

Architecture & Engineering

  • Build and evolve near Real Time ingestion frameworks using Pub/Sub, Kafka, Dataflow (Beam), GCS, BigQuery, Cloud SQL.
  • Architect fast, reliable ingestion pipelines with Apache Beam (Java/Python) on Dataflow.
  • Deliver robust CDC ingestion using Datastream, handling schema drift and low-latency data updates.
  • Apply deep BigQuery optimisation: Storage Write API, partitioning, clustering, materialised views.

Performance & Operations

  • Optimise streaming workloads for latency, throughput, backpressure, watermarks, and windowing.
  • Build self-healing, high-availability pipelines with enterprise-grade SLAs.
  • Establish strong observability: Cloud Monitoring, Logging, Trace, DLQs, replay strategies, retry frameworks.

Security, Governance & Leadership

  • Implement secure architectures across IAM, KMS, VPC-SC, DLP, Secret Manager.
  • Champion governance, data quality, and auditability.
  • Lead the NRT roadmap, mentor engineers, and partner with architecture and analytics teams.
  • Drive engineering excellence, reusable frameworks, and automation.

What You Bring

  • Deep hands-on expertise with GCP - Pub/Sub, Dataflow, BigQuery, GCS, Composer.
  • Strong background in CDC technologies (Datastream preferred).
  • Advanced Apache Beam engineering skills (Java or Python).
  • Solid grounding in Terraform, CI/CD, GitOps, and cloud-native orchestration (eg, GKE).
  • Proven leadership in designing and scaling complex streaming ingestion systems.

If you're an expert in Real Time data platforms and want to make a real impact - this is a career-defining opportunity.

Apply
Similar Jobs
Loading data...