Data Engineer

Job Description

The Data Engineer will join the Datastore Migration Factory team, responsible for migrating data and workloads from the on-premises Data Lake to an AWS-hosted Lakehouse environment. This is a high-visibility, business-critical project

1. Pipeline Migration

• Refactor and migrate extraction logic and job scheduling from legacy frameworks to the new Lakehouse environment.
• Execute data migration while ensuring data integrity and consistency.
• Collaborate with data owners and stakeholders to validate migrated assets and obtain sign-off.

2. Consumption Pattern Migration

• Convert and optimize legacy SQL and Spark-based workloads for compatibility with Snowflake and Apache Iceberg.
• Analyze data usage patterns to identify and deliver required data products.
• Coordinate with stakeholders to validate migrated workloads and meet business requirements.

3. Data Reconciliation & Quality

• Develop and execute data validation and reconciliation processes to ensure migrated data remains functionally equivalent to existing production datasets.
• Collaborate with data management platform teams and adapt to new workflows and technologies.

1. Basic Qualifications

• Education: Bachelor’s or Master’s degree in Computer Science, Applied Mathematics, Engineering, or a related quantitative field.
• Experience: 3–5 years of hands-on coding experience in a collaborative environment, with strong SQL troubleshooting and basic scripting skills.
• Programming: Proficiency in Python or Java.
• Development Practices: Strong understanding of SDLC, CI/CD best practices, and Kubernetes deployments.

2. Core Data Engineering Competencies

• Temporal Data Modeling: Managing historical data and state changes, including SCD Type 2.
• Schema Management: Schema evolution and enforcement, particularly with Apache Iceberg.
• Performance Optimization: Data partitioning and clustering.
• Data Architecture: Normalization, denormalization, and the use of natural and surrogate keys.

3. Technical Stack

• Extraction & Processing: Kafka, ANSI SQL, FTP, Apache Spark
• Data Formats: JSON, Avro, Parquet
• Platforms: Hadoop (HDFS/Hive), Snowflake, Apache Iceberg, Sybase IQ

Core Competencies

• Demonstrates integrity, accountability, and sound ethical judgment.
• Collaborates effectively across teams, functions, and global time zones.
• Communicates clearly and manages stakeholder expectations proactively.
• Takes ownership of deliverables and drives tasks to completion.
• Maintains urgency while ensuring quality and professionalism.
• Demonstrates curiosity, identifies risks early, and continuously seeks feedback and improvement.