Summary: The Senior Data Engineer role requires expertise in building and managing data pipelines using the Elastic Stack and Apache NiFi, with a focus on designing secure and scalable data solutions. The position is based onsite in Malvern and emphasizes compliance with strict security standards. The successful candidate will collaborate with various teams to ensure data quality and governance while managing large-scale data flows. This role offers the option of compressed hours and is classified as outside IR35.

Key Responsibilities:

• Design, develop, and maintain secure and scalable data pipelines using the Elastic Stack (Elasticsearch, Logstash, Kibana) and Apache NiFi.
• Implement data ingestion, transformation, and integration processes, ensuring data quality and security.
• Collaborate with data architects and security teams to ensure compliance with security policies and data governance standards.
• Manage and monitor large-scale data flows in real-time, ensuring system performance, reliability, and data integrity.
• Develop robust data models to support analytics and reporting within secure environments.
• Perform troubleshooting, debugging, and performance tuning of data pipelines and the Elastic Stack.
• Build dashboards and visualizations in Kibana to enable data-driven decision-making.
• Ensure high availability and disaster recovery for data systems, implementing appropriate backup and replication strategies.
• Document data architecture, workflows, and security protocols to ensure smooth operational handover and audit readiness.

Key Skills:

• Strong expertise in Elastic Stack (Elasticsearch, Logstash, Kibana) and Apache NiFi.
• Experience in designing and implementing data pipelines.
• Knowledge of data ingestion, transformation, and integration processes.
• Familiarity with data governance and security compliance.
• Ability to manage and monitor large-scale data flows.
• Proficiency in troubleshooting and performance tuning of data systems.
• Experience in building dashboards and visualizations.
• Strong documentation skills for data architecture and workflows.

Salary (Rate): £600

City: Malvern

Country: UK

Working Arrangements: on-site

IR35 Status: outside IR35

Seniority Level: Senior

Industry: IT