Data Engineer

Data Engineer

Posted 4 days ago by 1756535634

Apply
Negotiable
Outside
Hybrid
USA
Apply
Airflow Apache Airflow Apache Kafka Apache Spark Artificial Intelligence Automation Azure Databricks Azure Kubernetes Service BigQuery Cloud Computing Cloud Storage Cloud Technology Computer Science Containerisation Continuous Integration and Continuous Delivery Databricks Dataflow Data Infrastructure Data Integrity Data Modeling Data Quality Data Transformation Google Cloud Google Cloud Build Google Cloud Composer Google Cloud Platform (GCP) Infrastructure as Code (IaC) Jenkins Kubernetes Performance Tuning Probability And Statistics Python (Programming Language) SQL (Programming Language) Stakeholder Management Statistics Terraform Verification And Validation Workflow Automation Workflows

Summary: We are seeking a skilled Data Engineer to design, build, and maintain data pipelines and infrastructure on Google Cloud Platform. The ideal candidate will have expertise in BigQuery, Dataflow, and experience with AI/ML tools like Vertex AI and Gemini models. This role involves collaboration with data scientists and stakeholders to deliver effective data solutions. The position offers flexibility with remote or hybrid working arrangements in Atlanta, Georgia.

Key Responsibilities:

  • Build scalable batch and real-time data pipelines using Dataflow, Pub/Sub, Cloud Composer, and Dataproc.
  • Design and optimize analytical models in BigQuery, implementing best practices in schema design and performance tuning.
  • Deploy data infrastructure with Terraform; create and manage CI/CD pipelines for workflow automation.
  • Implement validation checks, ensure data integrity, and enforce security and governance practices.
  • Collaborate with data scientists to support Vertex AI workflows and explore the use of Gemini models for advanced data transformation.
  • Work with analysts, scientists, and business stakeholders to deliver impactful data solutions.

Key Skills:

  • Strong experience with Google Cloud Platform services: BigQuery, Dataflow, Pub/Sub, Cloud Storage.
  • Expertise in SQL, Python, and ETL/ELT development.
  • Knowledge of Infrastructure as Code tools (e.g., Terraform).
  • Familiarity with CI/CD tools: Google Cloud Build, Jenkins.
  • Understanding of data modeling, partitioning, clustering, and materialized views.
  • Working knowledge of data quality frameworks and governance principles.
  • Experience with Vertex AI, or a strong interest in ML/AI workflows on Google Cloud Platform.
  • Apache Spark, Kafka, Apache Airflow.
  • DBT or Dataform for transformations.
  • Docker, Kubernetes for Containerization.
  • Snowflake, Databricks or other modern platforms.

Salary (Rate): undetermined

City: Atlanta

Country: USA

Working Arrangements: hybrid

IR35 Status: outside IR35

Seniority Level: undetermined

Industry: IT

Detailed Description From Employer:

We are looking for a highly capable Data Engineer to design, build, and maintain reliable, scalable data pipelines and infrastructure on Google Cloud Platform (Google Cloud Platform).. The ideal candidate is proficient with BigQuery, Dataflow, and has experience integrating modern tools like Vertex AI and Gemini models for intelligent data workflows.

Core Responsibilities:

  • Data Pipeline Development: Build scalable batch and real-time pipelines using Dataflow, Pub/Sub, Cloud Composer, and Dataproc.
  • Data Warehousing: Design and optimize analytical models in BigQuery, implementing best practices in schema design and performance tuning.
  • Infrastructure & CI/CD: Deploy data infrastructure with Terraform; create and manage CI/CD pipelines for workflow automation.
  • Data Quality & Governance: Implement validation checks, ensure data integrity, and enforce security and governance practices.
  • AI/ML Integration: Collaborate with data scientists to support Vertex AI workflows and explore the use of Gemini models (via BigQuery ML or Vertex AI APIs) for advanced data transformation.
  • Cross-Team Collaboration: Work with analysts, scientists, and business stakeholders to deliver impactful data solution.

Technical Skills:

  • Strong experience with Google Cloud Platform services: BigQuery, Dataflow, Pub/Sub, Cloud Storage
  • Expertise in SQL, Python, and ETL/ELT development
  • Knowledge of Infrastructure as Code tools (e.g., Terraform)
  • Familiarity with CI/CD tools: Google Cloud Build, Jenkins
  • Understanding of data modeling, partitioning, clustering, and materialized views
  • Working knowledge of data quality frameworks and governance principles
  • Experience with Vertex AI, or a strong interest in ML/AI workflows on Google Cloud Platform
  • Apache Spark, Kafka, Apache Airflow
  • DBT or Dataform for transformations
  • Docker, Kubernetes for Containerization
  • Snowflake, Databricks or other modern platforms

Soft Skills:

  • Strong communication and collaboration skills
  • Ability to manage priorities and work independently
  • Analytical thinking and problem-solving mindset

Preferred Skills (Bonus):

  • Google Cloud certifications (e.g., Professional Data Engineer, ML Engineer)
  • Domain experience in energy, utilities, or industrial sectors

Experience Requirements:

  • Education: Bachelor s degree in computer science, Engineering, Statistics, or related technical field (or equivalent experience)
  • Professional Experience:
    • 6-8 years of experience in data engineering
    • Proven hands-on experience with Google Cloud Platform and modern data architecture

Apply
Similar Jobs
Loading data...