Day 2 - Batch vs Streaming

• Jobs with fixed schedules, like month-end settlement or daily revenue reports
• Reprocessing large volumes of historical data (backfill)
• Analyses where results can safely be minutes to hours late
• Preparing large-scale training data for machine learning models

When streaming fits:

• Real-time anomaly and fraud detection
• Live dashboards, real-time monitoring and alerting
• Immediate processing of IoT sensor data
• Recommendations and personalization that must react instantly to user behavior

The key point is that "real time is expensive." Streaming demands always-on infrastructure and more complex operations. So asking "do we really need real time, or is 5 minutes late acceptable" is the starting point of cost optimization. Vaguely deciding "fresher data is better, so stream everything" makes cost and complexity explode.

💡 Related theory: Micro-batch is the midpoint between the two. Like Spark Structured Streaming, it repeatedly runs small batches at very short intervals (a few seconds) to achieve "near real time." It is simpler to operate than true per-event streaming while achieving second-level latency, so many real-world pipelines choose this compromise.

The Latency-Throughput Trade-Off

These are the two metrics that quantitatively separate batch from streaming.

• Latency: the time from when data is generated until a result comes out. The lower it is, the closer to "real time."
• Throughput: the amount of data that can be processed per unit of time. The higher it is, the stronger the system is at "high volume."

Batch   : high latency ↑   very high throughput ↑↑↑
Stream  : low latency ↓    per-event processing → lower cumulative throughput

Batch collects data and processes it all at once, so per-record overhead is amortized and throughput is overwhelmingly high. In exchange, latency accrues for as long as you collect. Streaming processes each event the moment it arrives, so latency is very low, but every event carries processing and delivery costs, making the cumulative throughput achievable on the same infrastructure lower than batch.

On top of this comes the axis of correctness. Streaming must handle late-arriving data and out-of-order events, so it has to compromise between "accurate final results" and "fast approximate results." Batch processes after all the data has arrived, so it is free from this problem.

        Correctness ↑
          │
  Batch ● │
          │      ● Micro-batch
          │
          │            ● Streaming
          └──────────────────→ Freshness (low latency) ↑

💡 Related theory: The classic architecture-level solution to this trade-off is the Lambda Architecture. It runs a batch layer (accurate but slow) and a speed layer (fast but approximate) simultaneously, merging them in a serving layer. Because of the burden of maintaining two copies of the code, the Kappa Architecture — which handles even batch with a single streaming engine — emerged as an alternative. On AWS, the Kinesis/MSK + Flink combination corresponds to Kappa.

Mapping to AWS

This is the baseline for the exam's frequent "which service would you use for this scenario" questions.

In particular, keep two easily confused services straight. Kinesis Data Streams is for true streaming that needs low latency and multiple consumers, while Kinesis Data Firehose is a "managed near-real-time delivery service" that receives a stream and automatically loads it into S3, Redshift, and other destinations. If you just want data dropped at a destination without operating consumer code yourself, it's Firehose.

Wrapping Up

We established three judgment criteria today. First, batch processes bounded data periodically, while streaming processes an unbounded flow continuously. Second, the selection criterion is "how fresh do the results need to be," and since real time is expensive, don't default to streaming blindly. Third, batch is strong in throughput and correctness, streaming is strong in low latency — and micro-batch is the compromise between them.

In the next article, we unfold the full map of AWS data services that actually implement these processing paradigms, organized into ingestion, storage, processing, analytics, and governance categories.