Summary: We are looking for a seasoned Data Engineer with 7-8 years of experience in developing and optimizing data pipelines and managing data infrastructure. The role involves working with modern data technologies, particularly in cloud environments, to support data-driven decision-making. The ideal candidate should be a self-starter capable of working independently and collaboratively under tight deadlines. This position offers an opportunity to significantly impact data initiatives within the organization.

Key Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using Python, Spark, and PySpark.
• Build and optimize data marts and warehouses in Snowflake.
• Implement and manage workflows using Airflow for scheduling and orchestrating data pipelines.
• Leverage AWS services (e.g., EMR, S3, Lakehouse architecture) for cloud-based data platforms.
• Utilize GitLab CI/CD pipelines for automating testing, deployment, and monitoring of data solutions.
• Ensure high-quality data through robust validation, monitoring, and governance frameworks.
• Collaborate with stakeholders in credit card and retail domains to translate business requirements into technical solutions.
• Continuously monitor and optimize the performance of data pipelines and infrastructure.
• Maintain comprehensive documentation and advocate for industry best practices in data engineering.

Key Skills:

• Proficiency in Python for scripting and automation.
• Expertise in Spark and PySpark for big data processing.
• Hands-on experience with Airflow or other workflow orchestration tools.
• Advanced knowledge of Snowflake for data warehousing.
• Strong ETL/ELT skills with experience in building data marts.
• Experience with AWS services such as EMR, S3, and Lakehouse architecture.
• Familiarity with GitLab CI/CD pipelines for continuous integration and deployment.
• Knowledge of data quality frameworks, validation techniques, and governance practices.
• Business experience in credit card operations or retail industries is highly desirable.

Salary (Rate): undetermined

City: Edinburgh

Country: UK

Working Arrangements: undetermined

IR35 Status: undetermined

Seniority Level: undetermined

Industry: IT