Azure Data Engineer- (Databricks & Synapse)
Posted Today by Acumen Velocity
Negotiable
Undetermined
Remote
Remote
Summary: We are seeking an experienced Azure Data Engineer with strong expertise in Azure Databricks and Azure Synapse Analytics to support a government data modernization initiative. The role involves migrating legacy Informatica PowerCenter ETL workflows to scalable, cloud-native Azure platforms and ensuring compliance with government standards and processes.
Key Responsibilities:
- Migrate Informatica PowerCenter workflows, mappings, sessions, schedules, and dependencies to Azure Databricks and Azure Synapse Analytics
- Design and develop scalable ETL/ELT pipelines using Azure Synapse Pipelines and Databricks
- Implement data transformations using PySpark, Delta Lake, and Delta Live Tables
- Modernize legacy ETL architecture using cloud-native Azure services and distributed processing frameworks
- Optimize and manage distributed data processing using Spark (Spark UI, Databricks job logs)
- Ensure data quality, security, and compliance with government standards
- Collaborate with stakeholders, architects, and business teams to define data solutions and best practices
- Troubleshoot, debug, and enhance performance of data pipelines
Key Skills:
- Strong experience with Azure Databricks (PySpark, Delta Lake, Delta Live Tables)
- Experience with Azure Synapse Analytics (Pipelines, Notebooks)
- Proven experience in ETL migration from Informatica PowerCenter to Azure
- Strong knowledge of Apache Spark and distributed data processing
- Proficiency in Python/PySpark and SQL
- Experience with data pipeline orchestration and debugging tools
Salary (Rate): £75,000 yearly
City: undetermined
Country: undetermined
Working Arrangements: remote
IR35 Status: undetermined
Seniority Level: undetermined
Industry: IT
Job Summary:
We are seeking an experienced Azure Data Engineer with strong expertise in Azure Databricks and Azure Synapse Analytics to support a government data modernization initiative. The role involves migrating legacy Informatica PowerCenter ETL workflows to scalable, cloud-native Azure platforms and ensuring compliance with government standards and processes.
Key Responsibilities:
- Migrate Informatica PowerCenter workflows, mappings, sessions, schedules, and dependencies to Azure Databricks and Azure Synapse Analytics
- Design and develop scalable ETL/ELT pipelines using Azure Synapse Pipelines and Databricks
- Implement data transformations using PySpark, Delta Lake, and Delta Live Tables
- Modernize legacy ETL architecture using cloud-native Azure services and distributed processing frameworks
- Optimize and manage distributed data processing using Spark (Spark UI, Databricks job logs)
- Ensure data quality, security, and compliance with government standards
- Collaborate with stakeholders, architects, and business teams to define data solutions and best practices
- Troubleshoot, debug, and enhance performance of data pipelines
Required Skills:
- Strong experience with Azure Databricks (PySpark, Delta Lake, Delta Live Tables)
- Experience with Azure Synapse Analytics (Pipelines, Notebooks)
- Proven experience in ETL migration from Informatica PowerCenter to Azure
- Strong knowledge of Apache Spark and distributed data processing
- Proficiency in Python/PySpark and SQL
- Experience with data pipeline orchestration and debugging tools
Mandatory Requirement:
- Prior experience working on government/public sector projects
- Understanding of data security, compliance, and regulatory requirements in government environments
Preferred Qualifications:
- Experience with Azure DevOps (CI/CD pipelines)
- Familiarity with data warehousing and data lake architectures
- Experience working in Agile/Scrum environments