Summary: The Data Engineer role requires an experienced professional with 7-8 years in data engineering, focusing on building and optimizing data pipelines and managing data infrastructure. The candidate will work independently and collaboratively to support data-driven decision-making within the organization. Key responsibilities include developing ETL/ELT pipelines, optimizing data warehousing, and ensuring data quality. The position demands proficiency in modern data technologies and a strong ability to collaborate with business stakeholders.

Key Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using Python, Spark, and PySpark.
• Build and optimize data marts and warehouses in Snowflake.
• Implement and manage workflows using Airflow.
• Leverage AWS services to build and maintain cloud-based data platforms.
• Utilize GitLab CI/CD pipelines for automation of testing, deployment, and monitoring.
• Ensure high-quality data through validation, monitoring, and governance frameworks.
• Collaborate with stakeholders to understand business requirements and translate them into technical solutions.
• Monitor and optimize the performance of data pipelines and infrastructure.
• Maintain comprehensive documentation and advocate for best practices in data engineering.

Key Skills:

• Proficiency in Python for scripting and automation.
• Expertise in Spark and PySpark for big data processing.
• Hands-on experience with Airflow or other workflow orchestration tools.
• Advanced knowledge of Snowflake for data warehousing.
• Strong ETL/ELT skills with experience in building data marts.
• Experience with AWS services such as EMR, S3, and Lakehouse architecture.
• Familiarity with GitLab CI/CD pipelines for continuous integration and deployment.

Salary (Rate): undetermined

City: Edinburgh

Country: United Kingdom

Working Arrangements: undetermined

IR35 Status: undetermined

Seniority Level: undetermined

Industry: IT