All writing
January 8, 2025·5 min read

When Streaming Is Overkill (And When It Isn't)

Kafka and Flink are powerful, but most data problems don't need them. A practical framework for deciding when streaming infrastructure is actually justified.

  • Data Engineering
  • Kafka
  • Architecture
  • Batch

I built AthleteOS with Kafka and Apache Flink. I built the financial transaction pipeline with n8n polling Gmail every few minutes. Both are "real-time" in the sense that they process data as it arrives — but they're architecturally very different. Knowing which one to reach for is most of the decision.

The actual question

The question isn't "do I need real-time data?" It's "what is the cost of data being N minutes stale, and does that cost justify streaming infrastructure?"

For AthleteOS: the cost of pose data being 30 seconds stale during a workout is high — you're giving coaching feedback based on form from 30 seconds ago, which is useless. The Kafka + Flink investment is justified.

For the financial pipeline: the cost of transaction data being 5 minutes stale is zero. The finance team reviews reports once a day. A cron-triggered Airflow DAG polling for new emails every 5 minutes produces identical business value at a fraction of the operational complexity.

The complexity cost of streaming

Kafka + Flink introduces:

  • At-least-once vs exactly-once delivery semantics and the state management required to handle duplicates
  • Consumer group offset management — what happens when a consumer falls behind?
  • Schema evolution across producers and consumers (I use the Confluent Schema Registry)
  • Watermarking for out-of-order events — what if a WHOOP reading arrives 10 seconds late?
  • Backpressure handling when the processing layer can't keep up with ingestion rate
  • Separate monitoring and alerting (Kafka Consumer Lag + Flink job health vs just Airflow task status)

This is real operational overhead. For a single engineer working alone, each of those bullets is a potential 3am incident.

The micro-batch middle ground

Most "real-time" requirements are actually "low-latency batch." A dashboard that updates every 5 minutes isn't streaming — it's micro-batch. Airflow with a 5-minute schedule, Spark Structured Streaming with a 60-second trigger interval, or even a simple cron job covers this use case without Kafka.

The threshold I use: if your business requires data to be actionable within seconds and has a continuous high-volume event stream (thousands of events per minute), consider Kafka. If it requires data within minutes and has a moderate event rate, micro-batch is almost always the right call.

When I'd use streaming

  • Real-time recommendations or personalization where stale data degrades user experience directly (e-commerce, feeds, gaming)
  • Fraud detection where the detection window closes in seconds (card transactions, auth flows)
  • IoT sensor fusion where you're joining streams from multiple devices in real time
  • Event sourcing architectures where Kafka is the system of record, not just a transport layer

When I wouldn't

  • Analytics dashboards that update hourly or daily
  • ETL pipelines moving data between systems on a schedule
  • Webhook-triggered workflows where volume is low and irregular
  • Machine learning training pipelines (batch by nature)
  • Any pipeline where "real-time" means "within 5 minutes" — that's Airflow territory

The honest answer

Most data engineering work is batch. Most "real-time" requirements are low-latency batch. Kafka is genuinely useful for a specific class of problems that involve high-volume continuous event streams where seconds matter. Knowing which category your problem falls into before you architect saves weeks of work.

Build the simplest thing that meets the latency requirement. Optimize only when you can measure the cost of not doing so.