Senior Data Engineer

Role Overview

We are seeking experienced Data Engineers to lead and support the migration of large-scale analytical and reporting workloads from Amazon Redshift to Databricks Lakehouse Platform.

The ideal candidate will have hands-on experience in data warehouse modernization, ETL/ELT migration, performance optimization, cloud-native architectures, and big data processing.

Key Responsibilities

• Lead the migration of data pipelines, data models, and reporting workloads from Redshift to Databricks.
• Perform assessment and discovery of existing Redshift environments, schemas, workloads, and dependencies.
• Design and implement scalable data solutions using Databricks, Delta Lake, and Spark.
• Convert SQL-based ETL/ELT processes into Databricks notebooks, workflows, and pipelines.
• Optimize data ingestion, transformation, and storage for performance and cost efficiency.
• Develop batch and streaming data pipelines using PySpark and Spark SQL.
• Implement data quality, reconciliation, and validation frameworks during migration.
• Collaborate with business stakeholders, architects, and analytics teams to ensure successful migration and adoption.
• Troubleshoot performance bottlenecks and tune Spark jobs, Delta tables, and SQL workloads.
• Establish CI/CD processes and infrastructure-as-code practices for Databricks environments.
• Document migration strategies, architecture, and operational procedures.

Required Skills

Strong experience with large-scale data warehousing and data lake architectures.

Hands-on experience with Amazon Redshift and Databricks.

Expertise in PySpark, Spark SQL, and Python.

Strong SQL development and query optimization skills.

Experience with data modeling (Star Schema, Snowflake Schema, Dimensional Modeling).

Migration Experience

• Prior experience migrating from:
• Redshift to Databricks
• Teradata/Snowflake/Hadoop to Databricks (preferred)
• Data validation, reconciliation, and cutover planning.
• Performance tuning and workload modernization.