Senior Data Engineer -W2

Responsibilities

• Design, develop, and maintain scalable Big Data applications, ETL/ELT pipelines, and enterprise data integration solutions.
• Develop large-scale data processing solutions using Databricks, Apache Spark, PySpark, Python, Hadoop/HDFS, Hive, and Azure Data Factory (ADF).
• Build and optimize Databricks notebooks, Spark jobs, Delta Lake tables, and Lakehouse data processing workflows.
• Design and develop data ingestion, transformation, and integration pipelines using Azure Data Factory (ADF), including pipelines, activities, triggers, and dependencies.
• Develop complex SQL queries, stored procedures, data transformations, and source-to-target mapping logic.
• Build and maintain ETL workflows using SSIS and modern cloud-based data integration technologies.
• Develop reusable Python, PySpark, and Unix/Linux Shell Scripts for data processing, automation, validation, and operational support.
• Integrate data from relational databases, APIs, files, cloud storage, and enterprise applications.
• Develop and manage batch processing, job scheduling, dependencies, and workflow orchestration using Control-M, ADF, and Databricks Workflows.
• Implement data quality checks, reconciliation, exception handling, logging, and error recovery mechanisms.
• Perform performance tuning and optimization of Spark jobs, Databricks workloads, SQL queries, and data pipelines.
• Collaborate with data architects, business analysts, QA teams, and engineering teams to deliver scalable enterprise data solutions.
• Participate in Agile/Scrum activities, including sprint planning, refinement, daily stand-ups, reviews, and retrospectives.
• Support production deployments, troubleshooting, incident resolution, and post-deployment verification while ensuring established performance and quality standards are met.

Essential Qualifications:

• 8+ years of experience in Big Data Development, Data Engineering, ETL/ELT development, and enterprise data integration.
• Strong hands-on development experience with Python, PySpark, Apache Spark, SQL, and Databricks.
• Experience designing and developing large-scale data processing applications and ETL/ELT pipelines.
• Hands-on experience with Azure Data Factory (ADF) for data ingestion, transformation, integration, and workflow orchestration.
• Experience with Hadoop ecosystem technologies, including Hadoop/HDFS and Hive.
• Strong SQL development skills, including complex queries, stored procedures, transformations, and performance optimization.
• Experience developing and maintaining ETL workflows using SSIS.
• Experience with Unix/Linux Shell Scripting and enterprise job scheduling tools such as Control-M.
• Strong understanding of Delta Lake, Databricks Workflows, data warehousing, data modeling, and source-to-target data processing.
• Experience in data validation, reconciliation, error handling, production support, and performance tuning.
• Experience working in Agile/Scrum environments with strong analytical, troubleshooting, communication, and collaboration skills.

Desired Qualifications:

• Experience with Azure cloud services, including ADLS Gen2, Azure Synapse Analytics, and Azure DevOps.
• Experience with AWS services, including S3, Glue, EMR, Lambda, and Redshift.
• Knowledge of Kafka, Airflow, Java/Scala, Git, Jenkins, and CI/CD pipelines.
• Experience with Power BI, SSRS, Tableau, or other enterprise reporting and analytics tools.
• Prior experience supporting large-scale enterprise data platforms in healthcare, insurance, or other regulated industries.