Hadoop Engineer - ODP Platform
Posted 1 day ago by 1754986518
Negotiable
Undetermined
Hybrid
West Midlands
Summary: We are looking for a highly skilled Hadoop Engineer to enhance our Operational Data Platform (ODP) in an on-premises environment. The ideal candidate will possess extensive experience in the Hadoop ecosystem and strong programming skills, focusing on building and maintaining scalable data pipelines. This role requires collaboration with infrastructure and analytics teams to support operational data use cases while ensuring compliance with security and governance standards.
Key Responsibilities:
- Design, develop, and maintain data pipelines using Hadoop technologies in an on-premises infrastructure.
- Build and optimise workflows using Apache Airflow and Spark Streaming for real-time data processing.
- Develop robust data engineering solutions using Python for automation and transformation.
- Collaborate with infrastructure and analytics teams to support operational data use cases.
- Monitor and troubleshoot data jobs, ensuring reliability and performance across the platform.
- Ensure compliance with enterprise security and data governance standards.
Key Skills:
- Minimum 5 years of experience in Hadoop and data engineering.
- Strong hands-on experience with Python, Apache Airflow, and Spark Streaming.
- Deep understanding of Hadoop components (HDFS, Hive, HBase, YARN) in on-prem environments.
- Exposure to data analytics, preferably involving infrastructure or operational data.
- Experience working with Linux systems, shell scripting, and enterprise-grade deployment tools.
- Familiarity with monitoring and logging tools relevant to on-prem setups.
Salary (Rate): undetermined
City: Birmingham
Country: United Kingdom
Working Arrangements: hybrid
IR35 Status: undetermined
Seniority Level: undetermined
Industry: IT
Role Title: Hadoop Engineer / ODP Platform
Location: Birmingham / Sheffield - Hybrid working with 3 days onsite per week
End Date: 28/11/2025
Role Overview:
We are seeking a highly skilled Hadoop Engineer to support and enhance our Operational Data Platform (ODP) deployed in an on-premises environment.
The ideal candidate will have extensive experience in the Hadoop ecosystem, strong programming skills, and a solid understanding of infrastructure-level data analytics. This role focuses on building and maintaining scalable, secure, and high-performance data pipelines within enterprise-grade on-prem systems.
Key Responsibilities:
- Design, develop, and maintain data pipelines using Hadoop technologies in an on-premises infrastructure.
- Build and optimise workflows using Apache Airflow and Spark Streaming for real-time data processing.
- Develop robust data engineering solutions using Python for automation and transformation.
- Collaborate with infrastructure and analytics teams to support operational data use cases.
- Monitor and troubleshoot data jobs, ensuring reliability and performance across the platform.
- Ensure compliance with enterprise security and data governance standards.
Required Skills & Experience:
- Minimum 5 years of experience in Hadoop and data engineering.
- Strong hands-on experience with Python, Apache Airflow, and Spark Streaming.
- Deep understanding of Hadoop components (HDFS, Hive, HBase, YARN) in on-prem environments.
- Exposure to data analytics, preferably involving infrastructure or operational data.
- Experience working with Linux systems, shell scripting, and enterprise-grade deployment tools.
- Familiarity with monitoring and logging tools relevant to on-prem setups.
Preferred Qualifications:
- Experience with enterprise ODP platforms or similar large-scale data systems.
- Knowledge of configuration management tools (e.g., Ansible, Puppet) and CI/CD in on-prem environments.
- Understanding of network and storage architecture in data centers.
- Familiarity with data security, compliance, and audit requirements in regulated industries.
