Lead Data Engineer - AWS - Python - SQL - SC Eligible

The Role:

• Leading and line managing a team of data engineers, and growing their skills through pairing, review and coaching
• Designing how new source systems are integrated into the platform, from batch database extracts to API feeds
• Setting engineering standards for testing, CI/CD, documentation and data quality, and making sure they're followed
• Making architecture decisions across S3, Athena, Glue, Iceberg, dbt and Airflow, and writing them down so others can follow the reasoning
• Owning infrastructure as code in Terraform and keeping deployments repeatable
• Agreeing scope, priorities and delivery dates with product owners, system owners and senior data users
• Replacing manual or fragile pipelines with automated ones that can be monitored
• Representing data engineering in wider architecture and governance forums

Lead Data Engineer - AWS - Python - SQL - SC Eligible Key Skills:

• A track record of leading data engineering teams and making technical decisions that held up
• Expert Python and SQL
• Deep AWS experience: S3, Athena, Glue, IAM and related data services
• Production experience with dbt and Airflow, or close equivalents
• Strong software engineering practice: pytest, git, GitHub Actions or similar CI/CD, and infrastructure as code (Terraform)
• Experience integrating complex operational systems into an analytics platform
• Confidence managing stakeholders up to senior level, including pushing back when needed
• Eligible for SC clearance

Nice to have:

• Apache Iceberg, the medallion (bronze/silver/gold) pattern, or Write-Audit-Publish
• Docker, Kubernetes or Spark
• Line management in a consultancy or client-facing setting
• Experience in government or another highly regulated sector, including data protection and audit requirements