Kafka Engineer

Key Responsibilities

Lead Streaming & Real-Time Data Initiatives Collaborate with senior client stakeholders and strategic partners across Banking and Financial Services to architect, deploy, and scale Apache Kafka / Confluent Platform clusters for high-throughput, low-latency, fault-tolerant workloads. Design event-driven and streaming data architectures (event sourcing, CQRS, CDC, real-time ETL) aligned to banking use cases such as payments, fraud detection, AML, risk, and regulatory reporting. Conduct assessments, workshops, and capability presentations; respond to technical RFP components and shape streaming and real-time transformation programs.

Architect Scalable Streaming Infrastructure Implement and optimize Kafka Streams, ksqlDB, Kafka Connect, and Schema Registry, including exactly-once semantics where required. Design reusable frameworks for data ingestion, transformation, and movement using Kafka and complementary streaming tools (Spark Streaming, Apache Flink). Advise on managed versus self-managed trade-offs across Confluent Cloud, Amazon MSK, and self-hosted deployments, and design multi-region / DR architectures (MirrorMaker 2, Cluster Linking).

Drive Data Governance, Security, and Reliability Establish schema evolution, data contract standards, and governance across producers and consumers. Lead security and compliance hardening: encryption in transit and at rest, mTLS, RBAC/ACLs, audit logging, and alignment with financial regulatory and data-residency requirements. Build observability and SRE practices around the platform, including monitoring, alerting, capacity planning, and disaster recovery.

Engage with Senior Tech Stakeholders Understand enterprise goals, identify real-time and automation opportunities, and propose cutting-edge streaming solutions that integrate with broader data and AI/ML workflows. Troubleshoot production incidents consumer lag, rebalancing, partition skew, broker performance and drive root-cause resolution. Mentor client engineering teams and document architecture, runbooks, and best practices for long-term ownership.

Skills & Attributes for Success

Deep Kafka & Distributed Systems Knowledge - 7+ years in data engineering / distributed systems, with 4+ years of deep, production Apache Kafka experience. Expert knowledge of Kafka internals: partitioning, replication, consumer groups, ISR, offset management, and delivery semantics. Strong hands-on experience across the ecosystem: Kafka Connect, Kafka Streams, ksqlDB, and Schema Registry (Avro / Protobuf / JSON Schema).

Cloud and Big Data Expertise - Proven architecture experience across AWS, Azure, or Google Cloud Platform, with proficiency in containers (Docker) and orchestration (Kubernetes). Experience integrating Kafka with downstream systems such as databases, data lakes/warehouses, and microservices. Familiarity with the broader streaming and big data landscape, including Spark Streaming and Apache Flink.

Real-Time Engineering and Performance Proven experience operating Kafka at scale in production, including performance tuning, benchmarking, and incident response. Skilled in optimizing streaming platforms for throughput, latency, scale, and cost.

Data Security and Governance - Strong knowledge of implementing data security at rest and in motion, encryption standards, access controls, and compliance ideally within regulated financial environments (e.g., PCI-DSS, SOX, GDPR, data residency).

Client Engagement and Delivery Leadership - Ability to bridge the gap between business needs and technical solutions, ensuring alignment on architecture, scalability, security, and outcomes. Excellent communication and stakeholder-management skills; able to operate independently in a fully remote, client-facing consulting context.

Preferred / Nice-to-Have Direct experience in banking, payments, or financial services. Confluent certification (Developer or Administrator) and deep experience with Confluent Platform / Confluent Cloud. Experience with CDC tooling (Debezium), Infrastructure-as-Code (Terraform), and event-driven microservices. Proficiency in a JVM language (Java or Scala); comfort with Python a plus.