Data Engineer

Job Summary

We are looking for an experienced Data Engineer with strong expertise in Python, Microsoft Azure cloud services, and CI/CD pipelines. The ideal candidate will design, develop, and maintain scalable data pipelines, build reliable data integration solutions, and implement automated deployment processes to support enterprise data engineering initiatives.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Python.
• Build and optimize ETL/ELT workflows for processing structured and unstructured data.
• Develop cloud-based data engineering solutions using Microsoft Azure services.
• Work with Azure Data Factory, Azure Databricks, Azure Data Lake Storage, and Azure Synapse Analytics.
• Implement and manage CI/CD pipelines using Azure DevOps, Git, and automated deployment tools.
• Automate data processing, validation, testing, and deployment workflows.
• Optimize data pipelines for performance, scalability, reliability, and cost efficiency.
• Integrate data from APIs, databases, files, and other enterprise systems.
• Implement data quality checks, monitoring, logging, and error-handling mechanisms.
• Collaborate with data analysts, data scientists, architects, and development teams.
• Follow best practices for data security, governance, access control, and version management.

Required Skills

• Strong programming experience in Python.
• Hands-on experience with Microsoft Azure cloud services.
• Experience building ETL/ELT pipelines and data integration solutions.
• Strong understanding of SQL and relational databases.
• Experience with Azure Data Factory and Azure Data Lake Storage.
• Experience implementing CI/CD pipelines using Azure DevOps.
• Proficiency with Git, branching strategies, and version control.
• Knowledge of automated testing, deployment, and infrastructure management.
• Understanding of data modeling, distributed processing, and cloud architecture.
• Strong troubleshooting, analytical, and problem-solving skills.

Preferred Qualifications

• Experience with Azure Databricks and PySpark.
• Knowledge of Azure Synapse Analytics and Delta Lake.
• Experience with Docker, Kubernetes, or infrastructure as code.
• Familiarity with Terraform or ARM/Bicep templates.
• Experience working with REST APIs and event-driven data processing.
• Knowledge of data governance, security, and compliance practices.

Key Technical Skills

Python | Microsoft Azure | Azure Data Factory | Azure Databricks | PySpark | SQL | Azure DevOps | CI/CD | Git | ETL/ELT | Azure Data Lake Storage | Azure Synapse Analytics