Banking Fraud Detection Platform
Real-time fraud detection: CDC from Postgres, streaming through Kafka + Flink, indexed into OpenSearch for investigators.
Overview
Fraud detection has to be low-latency, resilient, and explainable. This project builds a production-shaped pipeline with CDC, event streaming, complex event processing, and a searchable case store.
Architecture
// interactive diagram coming next — click nodes to inspect
End-to-end data flow
- 01FastAPI accepts transactions and writes to PostgreSQL.
- 02Debezium streams row-level CDC events into a Kafka topic per table.
- 03Flink consumes CDC + reference data, applies CEP rules (velocity, geo, amount z-score).
- 04Alerts are produced back to Kafka and sunk into OpenSearch per-topic thread-safely.
- 05Investigators query cases in OpenSearch Dashboards; feedback loops relabel training data.
Hard problems solved
Kafka consumer rebalancing storms
Autoscaling Flink workers triggered repeated rebalances. Pinned partition assignment + cooperative-sticky assignor to bound movement.
CDC ↔ transaction ordering
Debezium snapshot + streaming handoff caused out-of-order events. Enforced key-based partitioning + Flink event-time watermarks with bounded lateness.
Dead-letter queues without silent drops
Bad payloads used to disappear. Every consumer now emits to a DLQ topic with error class + payload, replayed via a Flink job with a retry budget.
Thread-per-topic OpenSearch sink
One sink thread for many topics head-of-line-blocked alerts. Split into a thread-per-topic sink with bounded queues per index.
Lessons learned
- →Contracts (Avro + Schema Registry) are the cheapest insurance in streaming.
- →Event-time > processing-time for anything you want to audit.
- →DLQ + replay is not optional; it's part of the pipeline.