Data Engineer

The Opportunity

We're looking for an experienced SC Cleared Data Engineer to join a large-scale data platform programme responsible for collecting, processing, and analysing vast volumes of customer interaction data.

Working within a collaborative Agile team, you'll help build and maintain modern data products that support analytics, risk assessment, and operational decision-making. This role offers the opportunity to work with large-scale datasets, cloud-native technologies, and modern data engineering practices in a "you build it, you run it" environment.

Key Responsibilities

• Design, build, and maintain scalable data pipelines and data products.
• Process, validate, cleanse, and enrich large datasets from multiple sources.
• Develop and productionise data solutions using Python, Spark, and AWS technologies.
• Create and maintain Jupyter notebooks to support analysis and data engineering activities.
• Collaborate with multidisciplinary teams to deliver data-driven solutions.
• Support the publication, sharing, and consumption of data products.
• Maintain and improve automated data processing workflows.
• Champion testing, automation, reliability, and continuous improvement.

Essential Skills & Experience

• Strong commercial experience with:
• Python
• Spark
• SQL
• Database technologies
• AWS
• Experience building ETL and data transformation solutions.
• Strong understanding of data engineering best practices.
• Experience working with large-scale datasets.
• Knowledge of data modelling, validation, and data quality processes.
• Experience working within Agile delivery environments.
• Familiarity with CI/CD and automated deployment practices.
• Experience supporting and maintaining production data platforms.

Desirable Skills

• Scikit-Learn or Spark MLlib
• AWS EMR
• Apache Airflow
• Jenkins
• Hudi or other open table formats
• Kibana
• Grafana
• Understanding of Data Science principles