Summary: The Senior Data Engineer role focuses on building and managing data pipelines using the Elastic Stack and Apache NiFi, requiring DV clearance. The position involves designing, implementing, and maintaining secure data solutions while ensuring compliance with security standards. This is an onsite role based in Malvern, offering a 12-month contract with the possibility of compressed hours. The successful candidate will collaborate with various teams to ensure data quality and security throughout the data lifecycle.

Key Responsibilities:

• Design, develop, and maintain secure and scalable data pipelines using the Elastic Stack and Apache NiFi.
• Implement data ingestion, transformation, and integration processes, ensuring data quality and security.
• Collaborate with data architects and security teams to ensure compliance with security policies and data governance standards.
• Manage and monitor large-scale data flows in real-time, ensuring system performance, reliability, and data integrity.
• Develop robust data models to support analytics and reporting within secure environments.
• Perform troubleshooting, debugging, and performance tuning of data pipelines and the Elastic Stack.
• Build dashboards and visualizations in Kibana to enable data-driven decision-making.
• Ensure high availability and disaster recovery for data systems, implementing appropriate backup and replication strategies.
• Document data architecture, workflows, and security protocols to ensure smooth operational handover and audit readiness.

Key Skills:

• Apache NiFi
• DV Clearance
• Elastic Stack (Elasticsearch, Logstash, Kibana)
• Data pipeline design and management
• Data ingestion and transformation
• Data governance and security compliance
• Real-time data flow management
• Data modeling for analytics and reporting
• Troubleshooting and performance tuning
• Dashboard and visualization development in Kibana
• Disaster recovery and backup strategies
• Documentation of data architecture and workflows

Salary (Rate): £650

City: Malvern

Country: United Kingdom

Working Arrangements: on-site

IR35 Status: outside IR35

Seniority Level: Senior

Industry: IT