BigData Engineer

BigData Engineer

Posted Today by 1759915288

Apply
Negotiable
Outside
Remote
USA
Apply
Agile Methodology Algorithm Design Algorithms Amazon Web Services (AWS) Apache Hadoop Apache Hive Apache Spark Automation Azure Databricks Big Data Business Operations Cloud Computing Cloud Technologies Cloud Technology Computer Science Continuous Integration and Continuous Delivery Databricks Data Engineering Data Ingestion Data Pipeline Data Processing Distributed Computing Extract Transform Load (ETL) File Servers Git (Version Control System) Microsoft SQL Servers On Prem Pyspark Python (Programming Language) Regression Testing Scala (Programming Language) Scalability SQL (Programming Language) Sqoop

Summary: The BigData Engineer role involves building a highly functional Big Data platform to integrate data from various sources, enabling complex algorithm design for healthcare operations. The position requires expertise in AWS cloud technologies, ETL data pipeline development, and collaboration with internal teams to deliver data solutions. The engineer will also be responsible for maintaining application code quality and performance within Agile teams. This role is remote and classified as outside IR35.

Key Responsibilities:

  • Build a highly functional and efficient Big Data platform that brings together data from disparate sources and allows FinThrive to design and run complex algorithms providing insights to Healthcare business operations.
  • Build ETL Data Pipelines in AWS Cloud using AWS ADF and Databricks using PySpark and Scala.
  • Migrate ETL Data pipelines from On-Prem Hadoop Cluster to AWS Cloud.
  • Build Data Ingestion Pipelines in AWS to pull data from SQL Server.
  • Perform Automated and Regression Testing.
  • Partner with internal business, product and technical teams to analyze complex requirements and deliver solutions.
  • Participate in development, automation and maintenance of application code to ensure consistency, quality, reliability, scalability and system performance.
  • Deliver data and software solutions working on Agile delivery teams.

Key Skills:

  • Bachelor's degree in Computer science or a related discipline.
  • 6+ years of data engineering in an enterprise environment.
  • 6+ years of experience writing production code in Python, PySpark or Scala.
  • Strong knowledge of AWS platform. Should have worked in AWS ADF, Deployed ADF and Databricks code to production and be able to troubleshoot production issues.
  • Experience with SQL.
  • Experience with Big Data technologies in AWS such as Spark, Hive, Sqoop, Databricks or any other equivalent components.
  • Experience working with git and CI/CD tools.
  • Proven background in Distributed Computing, ETL development, and large-scale data processing.

Salary (Rate): undetermined

City: undetermined

Country: USA

Working Arrangements: remote

IR35 Status: outside IR35

Seniority Level: undetermined

Industry: IT

Detailed Description From Employer:

Job Description:

  • Build a highly functional and efficient Big Data platform that brings together data from disparate sources and allows FinThrive to design and run complex algorithms providing insights to Healthcare business operations.
  • Build ETL Data Pipelines in Aws Cloud using Aws ADF and Databricks using PySpark and Scala.
  • Migrate ETL Data pipelines from On Prem Hadoop Cluster to Aws Cloud.
  • Build Data Ingestion Pipelines in Aws to pull data from SQL Server.
  • Perform Automated and Regression Testing.
  • Partner with internal business, product and technical teams to analyze complex requirements and deliver solutions.
  • Participate in development, automation and maintenance of application code to ensure consistency, quality, reliability, scalability and system performance.
  • Deliver data and software solutions working on Agile delivery teams Requirements:
  • Bachelor's degree in Computer science or a related discipline
  • 6+ years of data engineering in an enterprise environment
  • 6+ years of experience writing production code in Python, PySpark or Scala
  • Strong knowledge of Aws platform. Should have worked in Aws ADF, Deployed ADF and Databricks code to production and be able to troubleshoot production issues.
  • Experience with SQL.
  • Experience with Big Data technologies in Aws such as Spark, Hive, Sqoop, Databricks or any other equivalent components.
  • Experience working with git and CI/CD tools
  • Proven background in Distributed Computing, ETL development, and large-scale data processing
  • Travel: None.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Apply
Similar Jobs
Loading data...