Data Engineer - Google Cloud - GCP - Airflow - Cloud Composer - Python - Docker - IAC - Terraform

Responsibilities

• Build and maintain scalable data pipelines for BigQuery using Cloud Composer and Airflow;
• Provide observability and monitoring using Monte Carlo and Looker as well as operational tools such as NewRelic and Splunk, driving reliability, data quality, data quality and robustness;
• Build critical data quality checks to ensure that the data we process and provide to partner teams meets security and compliance requirements;
• Participate in code and technical design reviews;
• Scope and plan work together with product engineering, data science and the business
• Support other teams for all tasks related to data tracking, collection, ingestion, quality monitoring and alerting.

Requirements

• 3+ years of experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt;
• Working knowledge of Python, SQL and Shell Scripting;
• Experience with Docker/k8s and infrastructure as a code tools such as Terraform, Salt or similar;
• 3+ years of significant professional experience developing applications written in a modern programming environment - preferably Scala/Java and/or Python;
• Professional experience across the entire software development life cycle, from gathering requirements to production deployment using CI & CD;
• Confident at articulating complex technical content to peers and partners across the organization via Technical Design Documents.