Data Engineer - Google Cloud - GCP - Airflow - Cloud Composer - Python - Docker - IAC - Terraform
Summary: The Data Engineer role focuses on developing and maintaining scalable data pipelines to support AI/ML user experiences and ensure robust product performance. The position requires collaboration with partner teams to deliver high-quality data while adhering to security and compliance standards. The ideal candidate will have extensive experience with GCP and related technologies, as well as a strong background in software development. This role is critical for driving data reliability and operational excellence within the organization.
Key Responsibilities:
- Build and maintain scalable data pipelines for BigQuery using Cloud Composer and Airflow;
- Provide observability and monitoring using Monte Carlo and Looker as well as operational tools such as NewRelic and Splunk, driving reliability, data quality, data quality and robustness;
- Build critical data quality checks to ensure that the data we process and provide to partner teams meets security and compliance requirements;
- Participate in code and technical design reviews;
- Scope and plan work together with product engineering, data science and the business;
- Support other teams for all tasks related to data tracking, collection, ingestion, quality monitoring and alerting.
Key Skills:
- 3+ years of experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt;
- Working knowledge of Python, SQL and Shell Scripting;
- Experience with Docker/k8s and infrastructure as a code tools such as Terraform, Salt or similar;
- 3+ years of significant professional experience developing applications written in a modern programming environment - preferably Scala/Java and/or Python;
- Professional experience across the entire software development life cycle, from gathering requirements to production deployment using CI & CD;
- Confident at articulating complex technical content to peers and partners across the organization via Technical Design Documents.
Salary (Rate): £350 per day
City: London
Country: UK
Working Arrangements: undetermined
IR35 Status: inside IR35
Seniority Level: undetermined
Industry: IT
Detailed Description From Employer:
Data Engineer - Google Cloud - GCP - Airflow - Cloud Composer - Python - Docker - IAC - Terraform
My client who are leaders in their field are looking for a Data Engineer who will own engineering efforts to provide partner teams within the organisation with data sources that build our AI/ML user experience and help run the client's product robustly and at scale.
Responsibilities:
- Build and maintain scalable data pipelines for BigQuery using Cloud Composer and Airflow;
- Provide observability and monitoring using Monte Carlo and Looker as well as operational tools such as NewRelic and Splunk, driving reliability, data quality, data quality and robustness;
- Build critical data quality checks to ensure that the data we process and provide to partner teams meets security and compliance requirements;
- Participate in code and technical design reviews;
- Scope and plan work together with product engineering, data science and the business
- Support other teams for all tasks related to data tracking, collection, ingestion, quality monitoring and alerting.
Requirements:
- 3+ years of experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt;
- Working knowledge of Python, SQL and Shell Scripting;
- Experience with Docker/k8s and infrastructure as a code tools such as Terraform, Salt or similar;
- 3+ years of significant professional experience developing applications written in a modern programming environment - preferably Scala/Java and/or Python;
- Professional experience across the entire software development life cycle, from gathering requirements to production deployment using CI & CD;
- Confident at articulating complex technical content to peers and partners across the organization via Technical Design Documents.
Similar Jobs
Apologies, there are currently no similar jobs available.