NiFi Data Engineer (SC Cleared)
Posted Today by Jobserve
Negotiable
Outside
Hybrid
London & South England
Summary: The NiFi Data Engineer role involves joining a collaborative development team to deliver a data Lakehouse architecture across cloud and on-premises platforms. The position requires hands-on data engineering skills, focusing on building and maintaining data pipelines while adhering to non-functional requirements. The successful candidate will work within Agile sprints and contribute to a fast-paced, security-conscious environment. This role is suitable for a self-starting engineer with experience in Lakehouse architectures and event-driven design.
Key Responsibilities:
- Design, develop and maintain data pipelines within a Lakehouse architecture across cloud and on-premises platforms
- Develop data models to support pipeline design, ensuring alignment with broader architectural standards and business requirements
- Handle non-functional aspects of pipeline development including security controls, performance optimisation and metadata labelling
- Implement event sourcing, change data capture and event-driven architecture patterns within the data platform
- Develop unit and integration tests for data pipelines as part of a CI/CD-integrated software development lifecycle
- Collaborate within an Agile development team across sprint planning, code review and iterative delivery
- Apply Java and Groovy within an object-oriented design approach to support pipeline development and tooling
Key Skills:
- Proven experience building data pipelines within Lakehouse architectures across cloud or on-premises environments
- Must be a UK national with SC clearance held and eligibility & willingness to undertake DV clearance
- Experience with event sourcing, change data capture and event-driven architecture patterns
- Familiarity with Java and Groovy, with a solid understanding of object-oriented design principles
- Experience working within an SDLC with an integrated development team using CI/CD tooling including Git
Salary (Rate): undetermined
City: London
Country: United Kingdom
Working Arrangements: hybrid
IR35 Status: outside IR35
Seniority Level: undetermined
Industry: IT
Detailed Description From Employer:
Job Title: NiFi Data Engineer
Location: Hybrid (up to 3 days per week on-site, across offices in London & South England)
Contract Length: 8 Months (with scope to extend)
Start Date: September 2026
IR35: Outside
Interview Process: 1 Stage, MS Teams
Clearance Required Before Applying: SC (with eligibility for higher levels of clearance)
We are supporting a public sector client in hiring a NiFi Data Engineer to join a collaborative development team delivering a data Lakehouse architecture across both cloud and on-premises platforms.
This is a hands-on data engineering role working within Agile sprints across a full software development lifecycle, building and maintaining data pipelines, developing data models and handling non-functional requirements including security, performance and metadata labelling. The successful candidate will work closely with engineers and architects to deliver robust, well-tested pipelines that support a modern Lakehouse architecture across AWS and on-premises environments.
This role suits a self-starting data engineer with experience in Lakehouse architectures, event-driven design and CI/CD-integrated development, who is comfortable picking up new technologies quickly and contributing from day one within a fast-paced, security-conscious environment.
Key Responsibilities
- Design, develop and maintain data pipelines within a Lakehouse architecture across cloud and on-premises platforms
- Develop data models to support pipeline design, ensuring alignment with broader architectural standards and business requirements
- Handle non-functional aspects of pipeline development including security controls, performance optimisation and metadata labelling
- Implement event sourcing, change data capture and event-driven architecture patterns within the data platform
- Develop unit and integration tests for data pipelines as part of a CI/CD-integrated software development lifecycle
- Collaborate within an Agile development team across sprint planning, code review and iterative delivery
- Apply Java and Groovy within an object-oriented design approach to support pipeline development and tooling
Essential Skills & Requirements
- Proven experience building data pipelines within Lakehouse architectures across cloud or on-premises environments
- Must be a UK national with SC clearance held and eligibility & willingness to undertake DV clearance
- Experience with event sourcing, change data capture and event-driven architecture patterns
- Familiarity with Java and Groovy, with a solid understanding of object-oriented design principles
- Experience working within an SDLC with an integrated development team using CI/CD tooling including Git
Desirable Skills
- Hands-on experience with Apache NiFi for data pipeline development and orchestration
- AWS experience including familiarity with EMR, S3 and SQS
- Awareness of domain-driven design approaches and architecture
- Linux awareness and Scripting skills
- PostgreSQL and PL/SQL knowledge