Senior Data Engineer

Senior Data Engineer

Posted 1 day ago by Response Informatics

Apply
Negotiable
Undetermined
Undetermined
Salford, England, United Kingdom
Apply
Azure Databricks Azure Data Factory Azure DevOps Continuous Integration and Continuous Delivery Databricks Data Engineering Data Manipulation Data Processing Data Quality DevOps Enterprise Architecture Generic Programming Git (Version Control System) Job Scheduling (Batch Processing) Metadata Microsoft Azure Pandas (Python Package) Python (Programming Language) Query Optimisation Scala (Programming Language) Scalability Scheduling Software Development Workflows

Summary: The Senior Data Engineer role involves designing, developing, and maintaining metadata-driven data pipelines using Azure Data Factory and Databricks. The position requires collaboration with cross-functional teams to integrate data solutions and implement CI/CD pipelines for automated deployment. The engineer will ensure data quality and governance while providing technical leadership on assigned projects. Strong expertise in data processing technologies and programming is essential for success in this role.

Key Responsibilities:

  • Design, develop, and maintain metadata-driven data pipelines using ADF and Databricks.
  • Build and implement end-to-end metadata frameworks, ensuring scalability and reusability.
  • Optimize data workflows leveraging SparkSQL and Pandas for large-scale data processing.
  • Collaborate with cross-functional teams to integrate data solutions into enterprise architecture.
  • Implement CI/CD pipelines for automated deployment and testing of data solutions.
  • Ensure data quality, governance, and compliance with organizational standards.
  • Provide technical leadership and take complete ownership of assigned projects.

Key Skills:

  • Azure Data Factory (ADF): Expertise in building and orchestrating data pipelines.
  • Databricks: Hands-on experience with notebooks, clusters, and job scheduling.
  • Pandas: Advanced data manipulation and transformation skills.
  • SparkSQL: Strong knowledge of distributed data processing and query optimization.
  • CI/CD: Experience with tools like Azure DevOps, Git, or similar for automated deployments.
  • Metadata-driven architecture: Proven experience in designing and implementing metadata frameworks.
  • Programming: Proficiency in Python and/or Scala for data engineering tasks.

Salary (Rate): undetermined

City: Salford

Country: United Kingdom

Working Arrangements: undetermined

IR35 Status: undetermined

Seniority Level: Senior

Industry: IT

Detailed Description From Employer:

Job description:

  • Key Responsibilities
  • Design, develop, and maintain metadata-driven data pipelines using ADF and Databricks.
  • Build and implement end-to-end metadata frameworks, ensuring scalability and reusability.
  • Optimize data workflows leveraging SparkSQL and Pandas for large-scale data processing.
  • Collaborate with cross-functional teams to integrate data solutions into enterprise architecture.
  • Implement CI/CD pipelines for automated deployment and testing of data solutions.
  • Ensure data quality, governance, and compliance with organizational standards.
  • Provide technical leadership and take complete ownership of assigned projects.

Technical Skills Required

  • Azure Data Factory (ADF): Expertise in building and orchestrating data pipelines.
  • Databricks: Hands-on experience with notebooks, clusters, and job scheduling.
  • Pandas: Advanced data manipulation and transformation skills.
  • SparkSQL: Strong knowledge of distributed data processing and query optimization.
  • CI/CD: Experience with tools like Azure DevOps, Git, or similar for automated deployments.
  • Metadata-driven architecture: Proven experience in designing and implementing metadata frameworks.
  • Programming: Proficiency in Python and/or Scala for data engineering tasks.
Apply
Similar Jobs
Loading data...