If you’ve ever had to pick between Kafka and MQTT for streaming data from the plant edge up to the cloud, you know it’s not a simple “one size fits all” answer. I’ve lived through this choice in real projects — from regulated environments to high-volume assembly lines — and the “right” answer always depends on the real constraints and goals on the ground. So, let’s get practical.
Where I’ve Seen MQTT Shine
Lightweight, Reliable, and Made for the Edge
Most of the time, when we’re wiring up PLCs, SCADA, or lab analyzers at the edge, the gateways (think: small industrial PCs or ruggedized boxes) don’t have a ton of horsepower. MQTT was built for exactly this — it’s a lightweight, publish/subscribe protocol that runs well even on resource-constrained gateways and unreliable networks. It’s also easy to secure (TLS, username/password, or certificates), and you can buffer messages during network hiccups (store-and-forward), so you don’t lose data if the WAN drops for a bit.
Unified Namespace: One Place for All Data
The biggest leap forward, in my view, was adopting a Unified Namespace (UNS) pattern using MQTT. Instead of every system talking point-to-point (which gets messy fast), everything publishes and subscribes to a central topic tree — usually structured by ISA-95 (Enterprise/Site/Area/Line/Cell). This made it way easier to onboard new equipment, standardize data models, and let multiple apps (MES, analytics, dashboards) subscribe to the same data in real time. Sparkplug B, layered on top of MQTT, adds features like auto-discovery, state awareness (“birth” and “death” messages), and standardized payloads — making integration and troubleshooting much easier.
MQTT’s Limitations
But MQTT isn’t perfect. It’s awesome for high-frequency, low-payload telemetry (like sensor values, status flags, or small events), but it’s not built for massive, durable event storage or replay. If you need to stream millions of records per second, or replay historical events for analytics, MQTT doesn’t have a built-in mechanism for that. You can get around this with persistent brokers and bridges, but it gets complicated fast.
Where Kafka Takes Over
High-Volume, Durable, and Analytics-Ready
When the goal shifts to handling huge volumes of data, with strong durability, replay, and integration with big data analytics or machine learning, Kafka becomes the backbone. Kafka is a distributed event streaming platform — not just a message broker — and it’s designed to ingest, store, and replay massive streams of data with high throughput and low latency. In practice, I’ve seen Kafka clusters handle millions of events per second, and you can replay any message from any point in time (as long as retention is configured), which is a lifesaver for advanced analytics, AI/ML, or regulatory audits.
But Kafka Is Heavy at the Edge
Here’s the honest truth: running Kafka directly on edge gateways is usually a bad idea. Kafka needs more CPU, memory, and disk than most industrial gateways can spare, and it’s not designed for intermittent or high-latency networks. Setting up, securing, and operating a Kafka cluster is a lot more work than spinning up an MQTT broker. For most edge-to-cloud scenarios, Kafka is best used in the cloud (or a big on-prem data center), not at the edge.
What Works Best: Hybrid Patterns
The “MQTT at the Edge, Kafka in the Cloud” Model
What’s worked best in my experience is combining both: MQTT for edge collection and real-time telemetry, Kafka for cloud-scale storage, analytics, and replay. Here’s how it usually looks:
- Edge Gateway: Collects data from PLCs, historians, lab devices (using OPC UA, Modbus, S7, etc.), buffers locally, and publishes via MQTT (often Sparkplug B) to a central broker.
- Cloud MQTT Broker: Receives data, can serve multiple subscribers (MES, dashboards, etc.).
- MQTT-to-Kafka Bridge: Bridges MQTT topics to a Kafka cluster in the cloud or data center.
- Kafka Cluster: Aggregates, stores, and streams data to data lakes (like Snowflake), AI/ML pipelines, and dashboards.
This lets you keep the edge simple and robust, while giving the enterprise all the power of Kafka for analytics, replay, and integration.
Table: When to Use What
| Scenario | MQTT | Kafka |
|---|---|---|
| Edge device resource constraints | Best fit | Not suitable |
| Flaky or low-bandwidth networks | Best fit | Not suitable |
| Real-time telemetry (small messages) | Best fit | Overkill |
| High-throughput, batch/event streaming | Not ideal | Best fit |
| Durable storage, replay, analytics | Limited | Best fit |
| Integration with cloud/AI/ML pipelines | Bridge needed | Native fit |
| Simplicity, fast deployment | Best fit | Complex |
Honest Opinion: Don’t Over-Engineer the Edge
If I could give one piece of advice: don’t try to run Kafka at the edge unless you have a really good reason (and a beefy infrastructure team). MQTT (with Sparkplug B and a good broker like HiveMQ) is more than enough for 99% of edge-to-cloud streaming needs. Only bring in Kafka when you need to aggregate, store, and analyze at scale in the cloud.
And — unpopular opinion — sometimes teams get obsessed with “future-proofing” and overcomplicate architectures. Start simple, get data flowing, and only add complexity when you hit real limits.
Final Takeaways
- Use MQTT at the edge for lightweight, reliable, real-time telemetry and UNS.
- Use Kafka in the cloud for high-throughput, durable event streaming, replay, and analytics.
- A hybrid approach (MQTT-to-Kafka bridge) gives you the best of both worlds.
- Don’t over-engineer: keep the edge simple, secure, and robust. Scale up in the cloud when you need to.
If you’re stuck deciding, focus on your real constraints: edge resources, network reliability, data volumes, and analytics needs. And if you want to see actual architectures or need help tuning a MQTT-to-Kafka bridge, I’m happy to discuss more real-world details.
Leave a Comment