Lead Data Engineer

Lead Data Engineer

Posted Today by Initialize IT

Apply
£500 Per day
Undetermined
Undetermined
Birmingham, UK
Apply
Apache Beam Apache Kafka Application Programming Interface (API) BigQuery Change Data Capture (CDC) Cloud Computing Cloud Technology Dataflow Data Ingestion Google Cloud Google Cloud Composer Google Cloud Platform (GCP) Google Kubernetes Engine (GKE) Java (Programming Language) Operational Excellence Python (Programming Language) Real Time Data SQL (Programming Language) Terraform

Summary: The Lead Data Engineer will be responsible for designing and developing Near Real Time data ingestion platforms on Google Cloud, focusing on low-latency ingestion pipelines. This role involves architecting solutions for high-throughput data flows into BigQuery and Cloud SQL while mentoring engineers and ensuring best practices. The position requires a strong technical leader to shape the roadmap and maintain the security and reliability of the ingestion platform.

Key Responsibilities:

  • Design and build near Real Time ingestion frameworks using Pub/Sub, Kafka, Dataflow (Beam), GCS, BigQuery, and Cloud SQL.
  • Develop and optimize streaming pipelines leveraging Apache Beam in Java or Python, deployed on Dataflow.
  • Implement CDC ingestion patterns using Datastream, ensuring resilience against schema drift and low-latency updates.
  • Utilize advanced BigQuery optimization techniques, including Storage Write API, partitioning, clustering, and materialized views.

Key Skills:

  • Deep hands-on expertise with Google Cloud Platform services including Pub/Sub, Dataflow, BigQuery, GCS, and Cloud Composer.
  • Strong experience with CDC technologies, especially Datastream.
  • Proficiency in Apache Beam (Java or Python) and SQL.
  • Solid knowledge of Terraform, CI/CD/GitOps, and container orchestration (eg, GKE).
  • Strong understanding of distributed systems, streaming design patterns, and cloud security controls.
  • Proven experience leading teams, driving technical strategy, and delivering complex ingestion systems at scale.

Salary (Rate): £500 a day

City: Birmingham

Country: UK

Working Arrangements: undetermined

IR35 Status: undetermined

Seniority Level: undetermined

Industry: IT

Detailed Description From Employer:

About the Role

We are seeking an experienced Lead Data Engineer to own and drive the design, development, and operational excellence of our Near Real Time (NRT) data ingestion platforms on Google Cloud. In this role, you will architect and deliver low-latency ingestion pipelines, enabling mission-critical, high-throughput data flows from diverse source systems into BigQuery and Cloud SQL using cutting-edge CDC and streaming technologies.

As a technical leader, you will shape the roadmap, mentor engineers, enforce best practices, and ensure our ingestion platform is secure, reliable, observable, and scalable.

Key Responsibilities

  • Design and build near Real Time ingestion frameworks using Pub/Sub, Kafka, Dataflow (Beam), GCS, BigQuery, and Cloud SQL.
  • Develop and optimize streaming pipelines leveraging Apache Beam in Java or Python, deployed on Dataflow.
  • Implement CDC ingestion patterns using Datastream, ensuring resilience against schema drift and low-latency updates.
  • Utilize advanced BigQuery optimization techniques, including Storage Write API, partitioning, clustering, and materialized views.

Required Qualifications

  • Deep hands-on expertise with Google Cloud Platform services including:

o Pub/Sub

    • Dataflow
    • BigQuery
    • GCS
    • Cloud Composer
  • Strong experience with CDC technologies, especially Datastream.
  • Proficiency in Apache Beam (Java or Python) and SQL.
  • Solid knowledge of Terraform, CI/CD/GitOps, and container orchestration (eg, GKE).
  • Strong understanding of distributed systems, streaming design patterns, and cloud security controls.
  • Proven experience leading teams, driving technical strategy, and delivering complex ingestion systems at scale.
Apply
Similar Jobs
Loading data...