Overview
Our client operates a cloud-based data platform that brings together trusted business information from a range of internal systems and applications.
As the platform's capabilities continue to expand, they are looking for an experienced Data Engineer to build the next generation of data pipelines, covering data provisioning, modelling, performance tuning and optimisation.
The role sits between IT development and business end-users, ensuring core datasets remain available, accurate and accessible, while also working with data and system owners to define access, security and data management processes.
The ideal candidate can take a data requirement end-to-end: examining a source data structure and building the extraction, storage and pipeline work needed to deliver it.
Key Responsibilities
- Design, build and maintain data pipelines from internal databases and SaaS applications into the client's cloud data platform
- Continuously develop and enhance the platform's data provisioning, modelling and performance capabilities
- Produce and maintain clear architecture and systems documentation
- Partner with business functions and other IT teams to scope data needs and deliver solutions
- Provide ongoing technical support, monitoring and troubleshooting for the data platform
- Act as a technical point of reference, sharing expertise and best practice across the team
Required Skills
- Degree qualified in a STEM discipline (or equivalent professional experience)
- Strong proficiency in a modern programming language (Python, Java, JavaScript, Scala, etc.)
- Strong SQL skills for querying and analysing structured data stores
- Proven experience designing, developing and deploying data pipelines
- Experience with a modern public cloud data platform (Azure, AWS or GCP)
- Experience with version control tooling
- Experience performing root-cause analysis of production incidents and debugging code defects
- Strong interpersonal and stakeholder-management skills
Desirable Skills
- Experience with Big Data toolkits and NoSQL technologies
- Experience with modern Business Intelligence toolkits
- Experience designing and documenting data models
- Experience optimising data processing workloads
- Experience with message queuing and stream processing
- Knowledge of modern data warehouse architectural principles
- Knowledge of DevOps workflows and principles
- Knowledge of Machine Learning/MLOps workflows and principles