Data Engineer

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Informatica, Azure Data Factory and Databricks.
• Process and transform structured and unstructured data from various sources into a centralized data lake or warehouse.
• Leverage Azure Data Lake Storage and Databricks Lakehouse to build efficient data storage and analytics solutions.
• Ensure seamless integration with Azure services such as Azure Event Hubs, Azure Stream Analytics, and Azure Functions.
• Optimize data pipelines for performance, scalability, and cost-effectiveness.
• Implement partitioning, indexing, and caching strategies to handle large-scale datasets.

Key Skills

• 5 years of experience as a data engineer or in a similar role.
• Hands-on expertise with Informatica, Azure Data Factory, Azure Synapse, and Azure Data Lake.
• Proficiency in Databricks for data transformation, analytics, and pipeline orchestration.
• Strong programming skills in Python, Scala, or SQL.
• Experience with distributed data processing frameworks (e.g., Apache Spark).
• Familiarity with CI/CD pipelines and version control systems (e.g., Azure DevOps, Git).
• Experience with Delta Lake and Databricks.
• Knowledge of real-time data ingestion and processing.
• Exposure to BI tools such as Power BI or Tableau.
• Strong problem-solving and debugging capabilities.
• Excellent communication and collaboration skills.
• Knowledge of data modeling techniques and schema designs for analytics.
• Preferred Azure / Databricks certifications.