Responsibilities
Designed, developed, and managed scalable cloud-based data platforms using Microsoft Azure Data Services, ensuring high availability, security, and performance.
Built and implemented Change Data Capture (CDC) solutions using Python, enabling real-time data synchronization between source and target systems with minimal latency.
Developed enterprise data warehouse solutions using Azure Synapse Analytics and Azure SQL Data Warehouse, optimizing reporting, analytics, and business intelligence workloads.
Wrote and optimized complex SQL queries, including CTEs, correlated subqueries, window functions, stored procedures, and joins to improve query performance and reduce execution time.
Developed scalable ETL/ELT pipelines using Azure Data Factory (ADF), Azure Databricks, and PySpark for ingesting structured and semi-structured data from multiple sources.
Processed and transformed large-scale datasets using Apache Spark (PySpark) on Azure Databricks, improving pipeline efficiency and processing performance.
Automated incremental, historical, and full-load processes using metadata-driven frameworks, event-based triggers, and control tables, reducing manual intervention by over 90%.
Designed robust Databricks notebooks with checkpointing, exception handling, logging, and restart capabilities to ensure reliable batch processing and data integrity.
Diagnosed and resolved Microsoft SQL Server performance bottlenecks, database corruption, deadlocks, and connectivity issues to maintain high system availability.
Designed and implemented dimensional data models, including Slowly Changing Dimensions (SCD Type 1 & Type 2) using Delta Lake MERGE operations to preserve historical data and improve data lineage.
Optimized Azure infrastructure costs through automated VACUUM jobs, storage lifecycle policies, cluster optimization, and workload migration strategies, significantly reducing operational expenses.
Managed large-scale distributed data using HDFS, improving ingestion, storage, retrieval, and transformation performance across distributed computing environments.
Implemented end-to-end data observability using Acceldata to monitor cluster health, resource utilization, pipeline execution, and performance regressions.
Developed Python scripts using Pandas and NumPy for data cleansing, validation, transformation, reconciliation, and quality checks.
Configured automated SMTP/Email notifications for pipeline failures, schema drift detection, SLA breaches, and operational alerts.
Used Linux shell scripting to automate file management, scheduling, and operational support for data engineering workflows.
Secured sensitive credentials and encryption keys using Azure Key Vault, ensuring compliance with enterprise security and governance standards.
Utilized GitHub for version control, code reviews, branching strategies, and CI/CD integration to support collaborative software development.
Participated in Agile/Scrum ceremonies, including sprint planning, backlog grooming, daily stand-ups, retrospectives, and release planning using JIRA and Confluence.
Collaborated with business stakeholders, QA teams, and data analysts during User Acceptance Testing (UAT) to validate business rules, data quality, and reporting accuracy.
Built interactive dashboards and analytical reports using Power BI and Tableau, enabling business users to derive actionable insights from enterprise data.
Integrated streaming and batch data pipelines using Kafka, Delta Lake, ADLS Gen2, and Azure Databricks to support modern Lakehouse architectures.