Data Engineer-Apache Nifi

Position Name: Senior Apache NiFi Data Engineer

Project Duration: 6 to 12 Months , Initially

Shift Details: US business hours (EST preferred)

Location/Mode -: WFH – Remote

Brief Job Description

Client is migrating closed trauma records from legacy registries (B5, TraumaBase and Trauma One formats) into a PostgreSQL data lake for reporting in Apache Superset.

The engineer will build Apache NiFi flows that pick up client-uploaded files from Azure Blob, load them through DuckDB into PostgreSQL, and make each new facility faster to onboard.

Flows are 80–90% similar within a format, so reusable templates are the core of the work.

EDI (835/837) ingestion is out of scope for now.

Key Responsibilities

• Design, build and maintain NiFi process groups that ingest, transform and load facility data end to end, keeping NiFi to two or three processors per load and pushing heavy parsing to DuckDB SQL.
• Write and optimize SQL to parse and flatten CSV, XML, JSON, Excel and DBF files, including unions across multiple files of one dataset.
• Implement CDC, incremental loads and idempotent upserts into PostgreSQL, following the client's existing trigger-based CDC tables.
• Handle file-encoding issues, skipped rows and schema differences between facilities, logging rejected rows for review.
• Build parameterized flow templates (Parameter Contexts, NiFi Registry) so new facilities onboard quickly across the 221-facility backlog.
• Add error handling, row-count reconciliation and Slack alerting to every pipeline.
• Tune NiFi and database performance: connection pools, batching and memory for large files.
• Version-control flows and SQL and support promotion across Dev, Test and Prod; support the NiFi 1.28.1 to 2.x upgrade if scheduled.
• Recommend improvements to credential handling and infrastructure set-up, and contribute to the client's living process document.

Good-to-Have Skills

• Healthcare integration: EDI X12 (837, 835, 270/271), HL7, Infor Cloverleaf (future scope only)
• NiFi scripting (Groovy or Python processors) or custom Java processor development
• NiFi 2.x and CI/CD for NiFi flow deployment
• Apache Superset and OpenMetadata (client's reporting and metadata tools)
• Talend experience (client's CDC tables mirror Talend's trigger-based CDC).