Role Descriptions:
Design, develop, and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
Build batch and streaming data pipelines using PySpark, Scala, and SQL.
Develop and manage Delta Lake-based data solutions on Azure.
Implement data ingestion and transformation processes using Azure Data Factory and ADLS.
Optimize Databricks jobs, Spark workloads, and data pipelines for performance and scalability.
Work with data engineers, analysts, and platform teams to deliver reliable data solutions.
Support analytics and ML workloads by preparing high-quality, secure, and reusable datasets.
Troubleshoot data pipeline issues and perform root-cause analysis.
Follow best practices for data security, monitoring, documentation, and deployment.
Contribute to CI/CD and production support activities for Azure data solutions.
Nice-to-Have Skills :
- Delta Lake
- Azure Synapse
- Azure DevOps / CI-CD
- Streaming technologies
- Data analytics / ML workloads
Must-Have Skills :
- Azure Data Factory (ADF)
- Azure Databricks
- Scala
- PySpark / Spark
- SQL
- Azure Data Lake Storage (ADLS)