The Default Answer Is Wrong
Ask almost any engineer today how to handle a low-latency data requirement and the answer comes back almost reflexively: stream it. Apache Kafka, Apache Flink, Spark Structured Streaming — the ecosystem is mature, the tutorials are plentiful, and the architectural pattern has acquired a kind of gravitational pull that makes it feel like the obvious choice. But obvious choices in software architecture are worth interrogating, and streaming is no exception. The decision to move from batch processing to a streaming architecture carries a set of real, often underappreciated trade-offs that can quietly erode the value of the system you set out to build.
This is not an argument against streaming. Streaming solves genuine problems and, in the right context, is the correct tool. The argument here is narrower: that the decision deserves more rigour than it typically gets, and that three categories of trade-off — operational cost, correctness complexity, and implicit latency guarantees — tend to be the ones that matter most and get discussed the least.
What You Are Actually Paying For
Batch processing has an unglamorous but significant advantage: it is cheap to operate. A batch job runs, finishes, and releases its resources. The infrastructure it needs can often be shared, scheduled during off-peak hours, or scaled down entirely between runs. Streaming infrastructure, by contrast, must stay alive continuously. The brokers need to be up. The consumers need to be running. The state stores, if you are doing anything stateful, need to be persisted and recoverable. All of that has a floor cost that does not go away when traffic is light.
This is not just a cloud billing concern, though that is real enough. It is also an engineering cost. Someone has to understand how the streaming system behaves under partition rebalancing. Someone has to know what happens when a consumer group falls behind and lag accumulates. Someone has to debug why exactly-once semantics produced a duplicate under a particular failure scenario that the documentation did not clearly anticipate. These are not exotic edge cases — they are the ordinary operational realities of running a streaming system in production. That expertise has to live somewhere in the organisation, and building it takes time.
Batch pipelines are not free to operate either, but their failure modes are generally more legible. A batch job failed at 3am: you have a clear start state, an end state, and a log. A streaming pipeline degraded gradually over six hours due to growing consumer lag and intermittent broker timeouts: reconstructing what happened and what data was affected is a substantially harder problem.
Correctness Is Harder Than It Looks
One of the most underestimated costs of streaming architectures is the complexity they introduce around correctness. In a batch world, correctness has a relatively clean definition: the job reads a bounded input, produces a bounded output, and you can verify both. If something goes wrong, you re-run the job against the same input and compare outputs. Idempotency is achievable with straightforward patterns.
In a streaming world, the input is unbounded and time is a first-class concern. This immediately opens up a set of problems that have no simple analogue in batch processing. Out-of-order events are not an edge case — they are the default, because networks are not synchronous and producers are not coordinated. Watermarking strategies that determine when a window can be safely closed involve genuine trade-offs between latency and completeness: close the window too early and you drop late-arriving data; hold it open too long and you introduce the latency you were trying to avoid in the first place.
Exactly-once processing guarantees, offered by several modern streaming platforms, are real but bounded. They typically apply within the streaming system itself. The moment your pipeline writes to an external sink — a database, an API, a downstream service — the guarantee only holds if that sink supports the necessary transactional semantics and you have configured the integration correctly. In practice, many sinks do not, and many integrations are not configured with this in mind. The result is a system that feels like it provides strong guarantees but delivers them only partially.
None of these problems are unsolvable. Experienced teams navigate them regularly. The point is that solving them correctly requires sustained investment in understanding the failure modes, and that investment should be factored into the architectural decision honestly rather than assumed away.
The Latency Promise You Did Not Mean to Make
Perhaps the most overlooked trade-off is what might be called the implicit SLA problem. When you build a streaming pipeline, you are not just choosing a processing model — you are making a promise to every downstream consumer of that pipeline about the latency they can expect. That promise does not need to be written down anywhere to be real. Once stakeholders observe that data arrives in near-real-time, their workflows, dashboards, and decisions begin to depend on that cadence. The expectation calcifies.
This matters because streaming systems can and do fall behind. Consumer lag under load, broker performance degradation, or a poorly tuned state backend can all cause a pipeline that normally delivers results in seconds to fall minutes or hours behind. In a batch system, a delay is visible and expected — the job has not run yet. In a streaming system, a delay can be invisible until someone notices that the dashboard showing "live" data is actually showing data from two hours ago. The operational and business impact of that silent degradation is often worse than an honest batch delay would have been.
More subtly, the decision to stream creates pressure to keep streaming. Migrating a downstream system that has built hard dependencies on low-latency data back to a batch model is politically and technically difficult, even if the streaming system turns out to be the wrong fit. The architectural decision has a kind of one-way quality that is worth recognising before it is made.
When Batch Is Still the Right Answer
There is a straightforward test worth applying before committing to a streaming architecture: what is the actual latency requirement, and who set it? Many systems that describe themselves as needing real-time data processing turn out, under examination, to need data that is fresh within five minutes, or within an hour, or simply before the start of the business day. These are not streaming requirements. A well-designed batch pipeline running on a short schedule can meet them with a fraction of the operational complexity.
The cases where streaming genuinely earns its overhead are the ones where latency is measured in seconds and that latency has direct business or user-facing consequences: fraud detection on a transaction that has not yet settled, a recommendation that needs to reflect an action the user took thirty seconds ago, an alerting system where a five-minute delay means a five-minute outage goes undetected. These are real use cases, and they justify the investment. The problem is that the streaming pattern has been adopted so widely that it is now being applied to problems that do not share these characteristics, importing the costs without capturing the benefits.
Making the Decision Deliberately
The right framework for this decision is not "streaming is modern, batch is legacy." It is a direct accounting of the three trade-offs described here. What is the true operational cost, in infrastructure and engineering expertise, of maintaining this pipeline in production? What are the correctness guarantees the system actually needs, and are you prepared to implement and verify them in a streaming context? And what latency SLA are you implicitly promising to downstream consumers, and can you meet it reliably under degraded conditions?
Streaming is a powerful and genuinely useful architectural pattern. It is not, however, a cost-free upgrade from batch processing. Treating it as one is how teams end up with systems that are expensive to run, difficult to reason about, and make promises they cannot always keep. The engineers and architects who get the most value from streaming tend to be the ones who chose it deliberately, with clear eyes about what they were trading away to get there.