Lead Data Engineer Data Migration

Role: Lead Data Engineer – Data Migration

Location: Remote (US)

Skills

Microsoft Fabric, DataStage, Azure, SQL, Python, PySpark/Apache Spark, Data Factory, ETL, Data Warehousing, Data Engineering, CI/CD

Key Responsibilities

• Lead the design, development, and deployment of scalable data pipelines and ETL workflows for DataStage to Microsoft Fabric migration.
• Develop and optimize pipelines using Microsoft Fabric Data Factory, OneLake, Lakehouse/Warehouse, and Apache Spark/PySpark.
• Design and implement data ingestion, transformation, integration, and migration solutions across structured and unstructured data sources.
• Perform source-to-target mapping and translate existing DataStage ETL logic into scalable Fabric-based solutions.
• Work with enterprise source systems including IBM DB2, IBM MQ, Snowflake, Oracle, Kafka, and other databases/messaging platforms.
• Develop complex data transformations and processing using SQL and Python/PySpark.
• Optimize data pipelines for performance, scalability, reliability, and cost efficiency.
• Implement data quality, governance, security, and monitoring best practices across the data platform.
• Lead code reviews, technical design discussions, troubleshooting, and resolution of complex data engineering issues.
• Collaborate with QA, Data Architects, BI Engineers, and business teams to deliver migration solutions aligned with functional and technical requirements.
• Support CI/CD and deployment automation using Azure DevOps and relevant orchestration tools.
• Mentor junior/mid-level data engineers and contribute to engineering standards, reusable frameworks, and technical documentation.

Required Experience

• 4-6 years of experience in data engineering, ETL development, data migration, or data warehousing.
• Strong hands-on experience with SQL, Python/PySpark, ETL, and data pipeline development.
• Experience with Microsoft Fabric/Azure, including Data Factory, OneLake, Lakehouse/Warehouse, or Synapse.
• Experience working with enterprise source systems such as DB2, IBM MQ, Snowflake, Oracle, or Kafka.
• Strong understanding of data modelling, data integration, performance optimization, and CI/CD.

Good to Have

• Prior experience with IBM DataStage, including DataStage jobs, transformations, and migration/modernization programs.
• Experience in DataStage-to-Fabric or similar ETL modernization projects.
• Exposure to Azure DevOps, Git, and automated deployment frameworks.