Senior Data Platform Engineer – Remote ( W2 Requirement)

Role Overview

Build, own, and operate a near real-time streaming data platform for operational data across the business, in a regulated environment.

Responsibilities

• Run the platform in production: own latency/throughput SLOs, monitoring, alerting, and runbooks; ensure idempotent, replayable pipelines, dead-letter triage, connector-failure recovery, and tuning for spiky, batch-driven claims load.
• Build and extend streaming pipelines that ingest CDC events from operational databases and SaaS sources into canonical, contract-validated form.
• Transform and enrich in Flink (Java and Flink SQL), including cross-stream joins that correlate events into unified lifecycle entities.
• Enable secure, governed data access over PHI: classification, row/column controls, and access policy applied as data is served to consumers (e.g., ABAC).
• Provision as code: define the platform (streaming, processing, storage, and catalog services on AWS) in CDK with CI/CD for data pipelines, and right-size for cost against the latency SLO.
• Partner across teams: work with upstream producers on source changes and contracts, downstream consumers on access and data needs (including self-service onboarding), and stakeholders to turn requirements into platform deliverables.

Qualifications

• 6+ years building and operating production data systems, including demonstrated ownership of streaming or event-driven pipelines — on-call, incident response, and recovery, not just development.
• Comfort debugging distributed pipelines (consumer lag, data skew, backpressure) with observability tooling (e.g., Datadog/CloudWatch).
• Production experience with Apache Kafka and Kafka Connect — partitioning and ordering guarantees, consumer groups, consumer-lag and broker-health troubleshooting, connector configuration and error handling, delivery guarantees (at-least-once, exactly-once/idempotent semantics), delete handling (tombstones, compaction), and replay/backfill under load.
• Production experience with a stateful stream processor, in Java — Apache Flink strongly preferred. Keyed state, checkpointing, event-time semantics (watermarks, windowing, late/out-of-order events), and deserialization/quarantine patterns.
• AWS-native data engineering across streaming, processing, storage, and catalog services (e.g., MSK, Glue, S3, Athena), with infrastructure-as-code — AWS CDK (preferred) or Terraform.
• Expert-level SQL and Python, and experience building and consuming REST APIs.
• AI-forward engineering mindset: hands-on use of AI-assisted and agentic development tools, an opinion on where AI adds value, and understanding of serving trusted operational data to agentic consumers as low-latency context, with lineage, permissions, and freshness.
• Change data capture and open table formats — log-based CDC (e.g., Debezium) and database internals behind it (e.g., replication slots, WAL, binlog across PostgreSQL and MySQL): snapshots and resnapshotting, heartbeats, offsets, and lag; plus Iceberg or Delta Lake partitioning and table maintenance.
• Data contracts, schema governance, and cataloging — contracts and schemas managed as code (versioned, CI-validated, compatibility-checked) on a schema registry (Avro, Protobuf, JSON Schema), with safe schema evolution end to end; familiarity with a technical metastore (e.g., Glue Data Catalog, Unity Catalog) and a governance/discovery catalog (e.g., Atlan, Alation, Collibra).
• Degree in Computer Science, Information Systems, or another quantitative field.
• Comfort on the command line / Unix-based OS (Mac + Linux environment).
• Health insurance domain knowledge — HIPAA Protected Health Information (PHI) and governing access to it.
• Excellent communication skills and demonstrated success in driving results through influence and collaboration.