Role Overview
We are seeking an experienced Senior DevOps Engineer to support and optimize a large-scale AWS-based data platform. The successful candidate will be responsible for maintaining and enhancing cloud infrastructure, CI/CD pipelines, automation frameworks, and platform reliability. This role requires a self-sufficient engineer who can work independently while providing technical guidance and mentoring to junior DevOps engineers employed by the client.
The position is primarily focused on operational support, platform optimisation, and continuous improvement. However, as the organization progresses with the decommissioning of Legacy platforms, there will be opportunities to design, build, and implement new platform components and automation capabilities.
Key Responsibilities
- Platform Support & Operations
- Support and maintain AWS-hosted data platforms and associated infrastructure.
- Monitor platform health, performance, security, and availability.
- Troubleshoot infrastructure, deployment, and application-related issues.
- Lead incident resolution and root cause analysis activities.
- Ensure platform services meet operational and service-level requirements.
Automation & DevOps Enablement
- Design, implement, and maintain CI/CD pipelines and deployment automation.
- Develop Infrastructure as Code (IaC) solutions using Terraform.
- Improve deployment reliability, consistency, and efficiency through automation.
- Maintain GitLab pipelines, runners, and supporting DevOps tooling.
- Create reusable automation frameworks and operational tooling using Python.
Platform Optimisation
- Identify opportunities to improve platform performance, reliability, security, and cost efficiency.
- Optimise AWS resources and data platform services.
- Support infrastructure upgrades, patching, and technical debt reduction initiatives.
- Contribute to platform standardisation and engineering best practices.
Technical Leadership
- Provide technical guidance and mentoring to junior DevOps engineers within the client team.
- Conduct code reviews and promote engineering standards and DevOps best practices.
- Collaborate with Data Engineers, Architects, Security teams, and Product stakeholders.
- Support knowledge transfer and capability development across the wider team.
Future Platform Development
- Participate in the design and implementation of new platform capabilities.
- Support migration and decommissioning activities for Legacy platforms.
- Contribute to platform architecture and strategic engineering initiatives.
Essential Skills & Experience
Technical Skills
Strong experience implementing and managing CI/CD pipelines and DevOps automation.
Proven Infrastructure as Code (IaC) experience using Terraform.
Advanced Python Scripting and automation experience.
Strong GitLab CI/CD and GitLab Runner administration experience.
Hands-on AWS experience, including:
IAM
EC2
S3
EMR
Glue
AMIs
CloudWatch
VPC
Security and access management
Experience supporting production cloud platforms and data workloads.
Strong understanding of DevOps principles, automation practices, and release management.
Experience troubleshooting complex operational and infrastructure issues.
Ability to work independently with minimal supervision.
Leadership & Collaboration
- Experience coaching and supporting junior engineers.
- Strong stakeholder management and communication skills.
- Ability to translate operational challenges into automation and improvement opportunities.
- Experience working within Agile delivery environments.
Desirable Skills
- Apache Spark administration or development experience.
- Data Engineering experience within AWS data ecosystems.
- Experience supporting data processing platforms such as EMR and Glue.
- PySpark experience.
- Knowledge of data lake and analytics architectures.
- Exposure to monitoring and observability tooling.
- Experience with platform modernisation and Legacy system decommissioning programmes.