Elastic Pipeline Engineer/Architect

Title: Elastic Pipeline Engineer

Duration: 3 Months

Location: Remote

ROLE SUMMARY

Hands-on delivery engineer responsible for building the automated redaction and evidence pipeline within the existing Elastic platform. Works under the technical direction of the Lead Architect, participating in discovery sessions and owning the development, configuration, and testing of the ingest-layer pipeline that automatically redacts sensitive fields at the point logs are ingested into Elasticsearch, applies tamper-evidence metadata to establish an immutable chain of custody, and enables auditor-facing self-service evidence retrieval.

KEY RESPONSIBILITIES

• Participate in discovery sessions alongside the Lead Architect and engineering and compliance stakeholders to map the current evidence lifecycle, identify manual touchpoints and custody gaps, and confirm the redaction ruleset
• Review the existing Elastic deployment to understand current index structure, ingestion patterns, access control configuration, and data retention policies, and evaluate candidate integration points at the Elastic ingest layer
• Develop and configure the automated redaction and evidence pipeline within the Elastic ingest layer in accordance with the approved solution design
• Implement field-level redaction rules dropping, masking, or hashing traceability fields per the data classification policy and unit-test them against representative pipeline output samples across all classified field types
• Validate that tamper-evidence metadata is correctly stamped at execution and preserved accurately within Elasticsearch to maintain an immutable chain of custody
• Configure Elasticsearch index templates, field mappings, and Index Lifecycle Management (ILM) policies aligned to audit and data retention requirements
• Configure auditor-facing search and retrieval views, implementing the role-based access model that distinguishes auditor views from operational engineering views
• Conduct integration and acceptance testing for evidence retrieval and data redaction against agreed acceptance criteria, documenting test results and remediation actions prior to handover
• Support UAT, capturing and resolving defects identified in the auditor retrieval workflow prior to engagement closure
• Contribute to pipeline architecture documentation, operational runbooks, and knowledge transfer sessions with the engineering and operations teams
• Participate in agile ceremonies daily stand-ups, sprint demos, and retrospectives

REQUIRED SKILLS & MINIMUM EXPERIENCE

Skill Min. Years

Elastic Stack Elasticsearch, ingest pipelines, index templates & ILM 5

Log pipeline / data transformation tooling (Logstash, ingest processors, Beats) 5

Python for data / integration automation 4

Data pipeline / ETL-style development 3

Working with structured & semi-structured data (JSON, schemas, field mappings) 3

Git / GitOps workflow 3

Data redaction, masking, or hashing techniques for sensitive fields 2

Role-based access control & search/retrieval configuration in Elastic (Kibana) 2

Working knowledge of data privacy / PII handling & data classification 2

CI/CD pipeline development (TeamCity, GitLab CI, Jenkins, or equivalent) 2

YAML / JSON data modelling 2

Linux / Bash 2

PREFERRED

• Prior experience delivering tamper-evident or chain-of-custody logging solutions in regulated environments
• Experience with Elastic security features field- and document-level security, role mapping, and audit logging
• Familiarity with CI/CD-driven change pipelines and pre-check / post-check validation outputs
• Experience integrating evidence or audit workflows for compliance and regulatory review
• Experience in regulated environments (financial services, GDPR)
• Client-facing professional services delivery experience