Change Data Capture Without Kafka: The Debezium and Postgres Pattern

What Is Change Data Capture, and Why Does It Matter?

Every production database is a living thing. Rows are inserted by user sign-ups, updated by background jobs, and deleted by retention policies — all day, every day. For a long time, the only practical way to react to those changes downstream was to poll the database on a schedule: query for rows where updated_at > last_run, process the results, repeat. It works, but it comes with a long list of problems. Polling misses hard deletes entirely. It imposes extra read load on a database that is often already under pressure. It introduces latency proportional to the polling interval. And it requires every table to carry a timestamp column, which is not always the case in legacy schemas.

Change Data Capture — CDC — takes a fundamentally different approach. Instead of asking the database what changed, CDC listens to the database's own internal record of what happened: the transaction log. In PostgreSQL, that log is called the Write-Ahead Log, or WAL. Every insert, update, and delete is written to the WAL before it is applied to the actual table, giving the database its durability guarantees. CDC tools tap into that stream and expose it to the outside world in near real-time. The result is a reliable, low-latency feed of every mutation, including hard deletes, without any polling and without requiring schema changes.

Enter Debezium: An Open-Source CDC Engine

Debezium is an open-source CDC platform built on top of Kafka Connect. It was designed, in its original form, to sit inside a Kafka Connect cluster, read the WAL from a source database, and publish change events to Kafka topics. This architecture is powerful and battle-tested, but it carries significant operational weight. Running Kafka Connect means running Kafka itself — a ZooKeeper ensemble or KRaft cluster, brokers, Connect workers, Schema Registry if you want Avro serialisation, and all the monitoring infrastructure that goes with it. For organisations that already operate Kafka at scale, this is a natural fit. For teams that do not, it can feel like installing a power plant to charge a phone.

That is where Debezium Server comes in. It is a standalone, self-contained runtime that embeds the same Debezium connectors but removes the Kafka dependency entirely. Instead of publishing change events to Kafka topics, Debezium Server writes them directly to a configurable sink — Redis Streams, Amazon Kinesis, Google Cloud Pub/Sub, Apache Pulsar, and others. The operational surface shrinks dramatically: a single process, a configuration file, and a target message broker. No Kafka, no Connect workers, no ZooKeeper.

How the PostgreSQL WAL Integration Works

To stream changes out of PostgreSQL, Debezium uses logical replication — a feature built into PostgreSQL that allows an external consumer to receive a decoded stream of database changes. The setup requires a few prerequisites on the PostgreSQL side. The wal_level parameter must be set to logical, which tells PostgreSQL to include enough information in the WAL for an external decoder to reconstruct row-level changes. A replication slot must be created, which is PostgreSQL's mechanism for ensuring the WAL is retained until the consumer has read it. And a publication must be defined, specifying which tables (or all tables) should be included in the stream.

Debezium connects to PostgreSQL using the replication protocol, consumes the logical replication stream, and translates the raw WAL data into structured change events. Each event contains a before image of the row (for updates and deletes), an after image (for inserts and updates), the operation type, the source table, and transaction metadata. These events are serialised — typically as JSON, though other formats are supported — and forwarded to the configured sink.

The replication slot is a critical piece of the puzzle, and it is also one of the most important operational concerns. PostgreSQL will not discard WAL segments that have not yet been consumed by a replication slot. If Debezium stops processing — because of a crash, a deployment, or deliberate maintenance — the replication slot keeps accumulating unconsumed WAL. On a busy database, this can fill the disk surprisingly quickly. Monitoring replication slot lag is not optional; it is essential.

Routing Change Events to Redis, Kinesis, or Pub/Sub

Once Debezium Server is capturing changes from PostgreSQL, the sink configuration determines where those events go. Each target has different characteristics that make it suitable for different use cases.

Redis Streams is a natural choice for teams already running Redis as part of their infrastructure. Change events are appended to a named stream, and consumers read from it using Redis's consumer group API. Because Redis keeps the stream in memory (with optional persistence), latency from capture to consumption can be very low. The trade-off is that Redis is not designed as a durable, long-term event store — retention is typically managed by capping stream length or setting expiry policies. This makes Redis Streams well-suited for use cases like cache invalidation, real-time notifications, or feeding a search index, where events are consumed quickly and long-term replay is not a requirement.

Amazon Kinesis is a managed streaming service designed for durable, scalable event ingestion at high volume. Routing Debezium change events to Kinesis makes the stream available to the broader AWS ecosystem — Lambda functions, Kinesis Data Analytics, Firehose for S3 delivery, and so on. Kinesis retains events for up to seven days by default, which provides a meaningful replay window. The shard-based partitioning model means that events from a given table can be consistently routed to the same shard, preserving order.

Google Cloud Pub/Sub offers a similar managed streaming experience within the GCP ecosystem. It is a push-pull message bus with at-least-once delivery semantics and configurable message retention. Pub/Sub integrates natively with Dataflow, BigQuery subscriptions, and Cloud Functions, making it a convenient entry point for CDC pipelines that ultimately land in analytical systems.

Operational Gotchas to Watch For

Debezium Server without Kafka simplifies the deployment, but it does not eliminate operational responsibility. Several failure modes deserve specific attention.

The replication slot lag problem bears repeating. If the Debezium Server process is interrupted and cannot reconnect — say, because the sink is unavailable and the retry budget is exhausted — the slot continues to hold WAL segments. Left unmonitored, this can cause disk exhaustion on the database host, which is a production-critical failure. A defensive approach is to set a maximum WAL size limit and alert aggressively on replication lag.

Snapshot behaviour is another area that surprises first-time users. When Debezium connects to a replication slot for the first time, it needs to establish a baseline view of existing data before it can stream incremental changes. By default, it performs an initial snapshot — a consistent read of the entire table set. On large databases, this snapshot can take a significant amount of time and generate substantial load. The snapshot strategy is configurable, and teams working with very large datasets should evaluate options like snapshotting only a subset of tables or using an existing backup to seed the initial state.

Schema changes also require careful handling. If a column is added or dropped on a source table, Debezium needs to be aware of the new schema to correctly parse subsequent WAL events. Debezium tracks schema history, and by default stores it in a Kafka topic — which is inconvenient when running without Kafka. Debezium Server supports alternative schema history implementations, including file-based storage, which should be configured explicitly before going to production.

Finally, exactly-once semantics are not guaranteed out of the box. Debezium provides at-least-once delivery: in a crash-recovery scenario, some events may be replayed. Consumers must be designed to be idempotent, using the event's source position or transaction ID as a natural deduplication key.

When This Pattern Makes Sense

The Debezium Server and PostgreSQL combination without Kafka is a strong fit for teams that need real-time data propagation but cannot justify the overhead of a full Kafka deployment. Microservice architectures that need to keep read models or caches in sync with a source-of-truth database, data pipelines that feed analytical stores in near real-time, and event-driven workflows triggered by database mutations are all well-served by this pattern. It is not a replacement for Kafka in high-throughput, multi-producer environments, but for the common case of a single PostgreSQL database feeding a downstream system, it offers a leaner, more maintainable path.

The open-source nature of Debezium means the community around it is active, and the connector's maturity with PostgreSQL logical replication is well-established. For teams evaluating CDC options, Debezium Server represents a meaningful reduction in operational complexity without sacrificing the core capability that makes change data capture valuable in the first place.