Apache Kafka and Event-Driven Architecture: Why Replaying History Is the Killer Feature
How Kafka's log-based publish-subscribe model, partitioning and consumer groups work, and why the ability to replay historical events is what genuinely separates it from a traditional message queue.
A log, not a queue
Traditional message queues like RabbitMQ or ActiveMQ implement a consume-and-delete model: a message is published, a consumer receives it, the consumer acknowledges it, and the queue deletes it. Once consumed, the message is gone. This model works well for the classic use case of distributing work items to be processed exactly once — a task queue for background jobs is the textbook example — but it structurally cannot support a second consumer coming along later and asking to see everything that happened, because the queue's job is precisely to make messages disappear once handled.
Kafka's foundational design decision is to model the message store not as a queue but as an append-only, immutable log, partitioned across a cluster of brokers, where messages (Kafka calls them records) are retained for a configurable period — commonly seven days, but frequently set to weeks, months, or indefinitely for critical event streams — regardless of whether any consumer has read them. Consumers do not remove records from the log when they read them; instead, each consumer independently tracks its own read position (its offset) within each partition, and can rewind that offset backward to reread history, or fast-forward to skip ahead, entirely independently of what any other consumer is doing. This single architectural choice — treat the message store as a durable, replayable log rather than a transient work queue — is the root of nearly every property that distinguishes Kafka from a traditional message broker, and it is worth being explicit that it is a genuine trade-off: Kafka's model requires more storage (you are retaining data other systems would discard) and pushes more responsibility for tracking read position onto the consumer, in exchange for capabilities a delete-on-read queue cannot offer at all.
Why replay is the killer feature
The practical payoff of the log-based model shows up whenever something downstream needs history that a delete-on-read queue would have already destroyed. If a new analytics team joins the company and wants to build a dashboard from six months of order events, a Kafka topic retaining that history lets them simply start a new consumer group and read from the beginning of the log — no need to have anticipated this consumer's existence when the events were originally produced. If a downstream consumer has a bug that corrupts its derived data for the last three days, the fix is to reset that consumer group's offset back three days and let it reprocess the same events from the log, correctly this time, rather than needing some separate backup-and-replay mechanism bolted on afterward. If a company wants to onboard a brand-new service — a fraud-detection model, say — that needs to be trained and backtested against a year of historical transaction events before going live, the events are still sitting in the log (or in Kafka's long-term tiered storage) ready to be consumed at whatever pace the new service needs, entirely decoupled from the live production traffic still flowing through the same topic in real time.
This replayability is also what makes Kafka the natural backbone for event sourcing architectures, where the log of events is treated as the primary source of truth and any derived database view (a customer's current balance, an order's current status) is understood as a projection that can, in principle, always be rebuilt by replaying the event log from the start. A traditional message queue cannot support this pattern at all, because by the time you realise you need to rebuild a projection, the underlying messages that built it the first time are already gone.
Partitioning and ordering: the mechanism behind Kafka's throughput
Kafka achieves its high throughput by splitting each topic into multiple partitions, each of which is an independent, ordered log distributed across the broker cluster. A producer writing to a topic assigns each record to a partition, typically by hashing a record key (such as a customer ID or order ID), which guarantees that all records sharing the same key always land in the same partition and are therefore always read in the order they were written, relative to each other. This is an important and frequently misunderstood nuance: Kafka guarantees ordering within a partition, not across an entire topic. Two records with different keys landing in different partitions carry no ordering guarantee relative to each other, which is a deliberate trade-off — total topic-wide ordering would require funnelling every write through a single sequential log, which is exactly the throughput bottleneck partitioning exists to eliminate.
This is why key selection is one of the most consequential design decisions in a Kafka-based system: choosing to key records by customer ID ensures every event for a given customer is processed in the correct relative order by whichever consumer handles that partition, while a poor key choice (or no key, which causes round-robin partition assignment) sacrifices ordering guarantees an application might actually depend on. On the consumption side, consumer groups extend this same partitioning logic: multiple consumer instances can join a group and Kafka automatically divides the topic's partitions among them, so a topic with 12 partitions and a consumer group of 4 instances gives each instance 3 partitions to process independently and in parallel, and this assignment rebalances automatically if an instance crashes or a new one joins, giving Kafka both horizontal scalability of consumption and automatic failover without any application-level coordination logic.
Where Kafka fits, and where it does not
None of this makes Kafka a universal replacement for traditional queues, and understanding the trade-off matters for choosing correctly. A queue's delete-on-consume model is simpler to reason about for pure task-distribution workloads where nobody will ever need the history and storage cost genuinely matters, and traditional queues typically offer richer per-message routing semantics (priority queues, complex routing rules based on message content) that Kafka's simpler partition-and-consumer-group model does not directly provide. Kafka also introduces genuine operational complexity — running and tuning a distributed, partitioned, replicated log cluster is a heavier operational commitment than running a single RabbitMQ instance, and choosing partition counts, retention policies and key strategies badly can create problems (too few partitions caps parallelism; too many creates broker overhead; a bad key choice creates "hot partitions" where load is unevenly distributed across the cluster) that are hard to fix after a topic already has significant production traffic.
The pattern that has emerged across most large-scale data architectures is to use Kafka specifically as the durable, replayable backbone connecting many independent producers and consumers — the nervous system through which order events, clickstream data, sensor readings or change-data-capture streams flow to every system that needs them, including feeding directly into stream-processing frameworks like Kafka Streams or Flink for real-time aggregation, and into the vector databases, feature stores and data warehouses covered elsewhere in this kind of infrastructure stack — while reserving simpler message queues for narrower, ephemeral task-distribution needs where replay and multi-consumer fan-out were never going to be required in the first place.
Frequently Asked Questions
Does Kafka guarantee that all messages in a topic are processed in the order they were sent?
Only within a single partition. Records with the same key always land in the same partition and are read in order, but Kafka makes no ordering guarantee across different partitions of the same topic, which is a deliberate design trade-off that enables its parallel throughput.
How long does Kafka keep messages after they are consumed?
By default, based on a configurable time-based retention policy (commonly seven days) or size limit, regardless of whether any consumer has read the messages. Consumption does not delete records, which is what allows multiple independent consumers to read the same history at different times.
What happens if a consumer in a consumer group crashes?
Kafka automatically rebalances the group, reassigning the crashed consumer's partitions to the remaining active consumers in the group, so processing continues without manual intervention, resuming from each partition's last committed offset.
Is Kafka a database?
Not in the traditional sense, though the line has blurred with features like Kafka Streams' state stores and tiered storage for very long retention. Kafka is best understood as a durable, ordered, replayable log for event data, which many architectures use as the source of truth that databases and other derived views are built from.