Sr Data Engineer

Sr Data Engineer

Posted 4 days ago by 1764672865

Apply
Negotiable
Outside
Remote
USA
Apply
Airflow Amazon Web Services (AWS) Apache Airflow Apache Hadoop Apache Kafka Apache Spark AWS Elastic MapReduce (EMR) AWS Lambda Azure Databricks BigQuery Cloud Computing Cloud Technology Continuous Integration and Continuous Delivery Databricks Dataflow Data Quality Data Warehousing Google Cloud Google Cloud Platform (GCP) Informatica Machine Learning Microsoft Azure Python (Programming Language) Scalability Snowflake (Data Warehouse) SQL (Programming Language) Verification And Validation Workflows

Summary: The role of Sr Data Engineer involves designing, developing, and maintaining end-to-end data pipelines, optimizing data architectures for analytics and machine learning workflows. The position requires collaboration with data scientists and ML engineers, as well as managing ETL/ELT processes using various tools. The engineer will work with distributed systems and cloud platforms to ensure data quality and system performance. Strong skills in Python and SQL are mandatory for this position.

Key Responsibilities:

  • Design, develop, and maintain end-to-end data pipelines (batch & streaming)
  • Build and optimize data architectures for analytics, reporting, and ML workflows
  • Manage ETL/ELT using Airflow, dbt, SSIS, Informatica, Glue, Dataflow
  • Collaborate with Data Scientists & ML Engineers on model training and feature engineering
  • Work with distributed systems: Spark, Hadoop, Kafka, Databricks
  • Implement data quality frameworks and automate validation checks
  • Hands-on experience with AWS (S3, Glue, Redshift, EMR, Lambda) OR Azure (ADF, Synapse, Databricks) OR Google Cloud Platform (BigQuery, Dataflow, Dataproc)
  • Optimize data systems for performance, scalability, cost, and security
  • Maintain and enhance data warehouse platforms (Snowflake/BigQuery/Redshift)
  • Experience with Docker/Kubernetes & CI/CD is a plus
  • Strong skills in Python & SQL (mandatory)

Key Skills:

  • Experience in designing and maintaining data pipelines
  • Proficiency in ETL/ELT tools such as Airflow, dbt, SSIS, Informatica, Glue, Dataflow
  • Knowledge of distributed systems like Spark, Hadoop, Kafka, Databricks
  • Experience with cloud platforms: AWS, Azure, Google Cloud Platform
  • Strong skills in Python and SQL
  • Familiarity with data quality frameworks and validation checks
  • Experience with data warehouse platforms like Snowflake, BigQuery, Redshift
  • Experience with Docker/Kubernetes & CI/CD is a plus

Salary (Rate): £44.00 hourly

City: undetermined

Country: USA

Working Arrangements: remote

IR35 Status: outside IR35

Seniority Level: undetermined

Industry: IT

Detailed Description From Employer:

Data Engineer

Location: Remote, USA

Rate : $55/hr on W2

Email:

Job Description:

Design, develop, and maintain end-to-end data pipelines (batch & streaming)

Build and optimize data architectures for analytics, reporting, and ML workflows

Manage ETL/ELT using Airflow, dbt, SSIS, Informatica, Glue, Dataflow

Collaborate with Data Scientists & ML Engineers on model training and feature engineering

Work with distributed systems: Spark, Hadoop, Kafka, Databricks

Implement data quality frameworks and automate validation checks

Hands-on experience with AWS (S3, Glue, Redshift, EMR, Lambda) OR Azure (ADF, Synapse, Databricks) OR Google Cloud Platform (BigQuery, Dataflow, Dataproc)

Optimize data systems for performance, scalability, cost, and security

Maintain and enhance data warehouse platforms (Snowflake/BigQuery/Redshift)

Experience with Docker/Kubernetes & CI/CD is a plus

Strong skills in Python & SQL (mandatory)

Apply
Similar Jobs
Loading data...