IoT Streaming: How to Process Sensor Data in Real Time

IoT streaming is the continuous flow of telemetry from sensors, machines, vehicles, and connected products into systems that process it the moment it arrives. Instead of waiting for devices to upload files in batches, an IoT streaming pipeline filters, aggregates, and acts on each reading within seconds. This guide covers device and sensor data, not live video from IoT cameras.

What is IoT streaming?

Most early IoT projects collected data in batches. Devices buffered readings and uploaded them every hour or every night, and analysts looked at the results the next day. That works for reporting, but it fails when a compressor is overheating, a refrigerated truck is warming up, or a smart meter shows a sudden spike. IoT streaming treats every reading as an event and moves it through a pipeline that can react while the situation is still happening.

What makes IoT data different

IoT data is harder to process than most event streams. A single fleet can produce millions of small messages per minute, each carrying a few bytes of telemetry. Devices lose connectivity and then send hours of buffered readings at once, so events arrive late and out of order. Payload formats also drift, because a fleet rarely runs one firmware version, and older devices keep sending older schemas long after new ones ship.

These properties shape every design decision that follows. A pipeline that ignores them produces dashboards that look correct until the first network outage or firmware rollout.

How IoT streaming works

A typical IoT streaming architecture has four stages, and each one solves a different part of the problem. The fundamentals of the processing stage are covered in our guide to stream processing.

Devices and gateways

Sensors and devices publish readings directly or through a local gateway. MQTT is the most common protocol at this stage, because it is lightweight, runs on constrained hardware, and tolerates unreliable networks. MQTT version 5.0 has been an OASIS standard since March 7, 2019. In industrial settings, gateways often translate machine data from OPC UA or older fieldbus protocols before publishing it.

MQTT broker to Apache Kafka

An MQTT broker manages the device connections, which can number in the millions. It is not built to store long histories or to feed many downstream consumers independently, so most architectures bridge it to Apache Kafka, either through a Kafka Connect MQTT source connector or through a broker-side Kafka integration. Kafka stores every reading durably, lets new applications replay history, and decouples device connectivity from the systems that use the data.

Stream processing

A stream processing engine reads from Kafka and does the work that turns raw telemetry into usable signals. Apache Flink is a common choice for IoT because it handles event time, state, and late data natively. Kafka Streams fits when the processing logic lives inside a microservice, and Flink SQL lets data teams write the same logic as continuous queries.

Storage and serving

Processed data lands where people and applications can use it. Time-series databases serve operational dashboards and recent history, lakehouse tables keep the long-term record for analysis and model training, and alerting tools notify operators or trigger automated actions. Our guide to streaming analytics covers how teams turn these results into decisions.

How to process streaming data from IoT devices

The processing stage is where IoT pipelines succeed or fail. Five patterns come up in almost every project.

Normalizing and enriching payloads

The first job is to convert every device’s payload into one canonical format, whatever its firmware version. A schema registry and versioned schemas make this manageable. The pipeline then enriches each reading with device metadata, such as location, asset type, or customer, by joining the stream with a reference table that is itself kept current from the device registry.

Windowed aggregation and downsampling

Few consumers need every raw reading. Tumbling or hopping windows reduce one-second telemetry to one-minute averages, minimums, and maximums per device, which cuts storage cost and makes dashboards faster. Raw data can still go to cheap long-term storage for audits and model training.

Event time, watermarks, and late sensor data

When a device reconnects after an outage, its readings describe the past, not the moment they arrive. Processing by event time, the timestamp the device recorded, keeps aggregates correct. Flink uses watermarks to track how far event time has progressed, so an operator knows when a window can close even if records arrived out of order. Teams set an allowed lateness and route anything later to a side output for correction.

Device state and digital twins

Many questions depend on history per device, such as whether a temperature has risen for 10 consecutive minutes or whether a door has been open longer than usual. Keyed state in the stream processor holds the current picture for each device. Taken together, these per-device states form a live digital twin of the fleet that applications can query.

Anomaly detection and alerting

The final pattern compares live readings with thresholds, rules, or model predictions and raises alerts when they diverge. Simple rules catch obvious failures, while pattern detection across several events catches slower problems such as a bearing that vibrates a little more each day. Alerts should carry enough context, such as the device, location, and recent readings, for an operator to act without opening another system.

Edge vs cloud processing

Not every computation belongs in the cloud. The choice depends on latency, bandwidth, and what happens when the connection drops.

Most production systems combine both. The edge filters, downsamples, and handles time-critical control, and the cloud handles enrichment, fleet-wide correlation, and history.

MQTT vs Kafka for IoT

MQTT and Kafka are often compared, but they do different jobs and usually work together.

IoT streaming use cases

In fleet and logistics, vehicles stream location, engine data, and cargo temperature, and the pipeline flags delays, route deviations, or cold-chain breaches while drivers can still respond. In energy and utilities, smart meters and grid sensors stream consumption and voltage readings that feed load forecasting and outage detection.

Smart buildings stream occupancy, air quality, and equipment data to adjust heating and cooling and to schedule maintenance. Makers of connected products, such as appliances, heating systems, and medical devices, use the same pipeline to monitor installed units, plan service visits, and learn how customers use their products. For factory-floor use cases, see our guide to data streaming in manufacturing.

Retired IoT platforms to plan around

Several managed IoT services that older architecture guides recommend are gone or retiring. Google Cloud IoT Core was retired on August 16, 2023. AWS IoT Analytics stopped accepting new customers on July 24, 2024 and reached end of support on December 15, 2025. Microsoft has announced that Azure IoT Central will retire on September 20, 2029, and points customers to Azure IoT Hub, Device Provisioning Service, and Azure Device Registry, with Microsoft Fabric for analytics.

These retirements are a strong argument for keeping the streaming core portable. When device connectivity runs through standard MQTT and the event backbone runs on Kafka, a change of cloud service touches one layer instead of the whole pipeline.

Challenges in IoT streaming

IoT streaming projects tend to struggle in the same places:

• Device identity and security, including certificate rotation across large fleets
• Schema changes across firmware versions that break downstream consumers
• Data floods when thousands of devices reconnect at once after an outage
• Retention costs for raw telemetry that nobody reads after the first week
• End-to-end observability, so teams can tell a silent device from a broken pipeline

Each of these is easier to solve in the design phase than after the fleet has grown.

FAQs

Is IoT streaming the same as IoT video streaming?

No. IoT streaming usually refers to telemetry: small, frequent readings such as temperature, location, or vibration. IoT video streaming sends continuous camera footage and relies on different protocols and infrastructure. Some pipelines combine the two, for example by streaming events extracted from video analysis alongside sensor data.

How do you handle late or out-of-order sensor data?

Process by event time, the timestamp the device recorded, rather than by arrival time. A stream processor such as Apache Flink uses watermarks to decide when a time window is complete, and an allowed-lateness setting keeps windows open for stragglers. Readings that arrive later still can go to a separate stream that corrects stored results.

Which stream processing engine is best for IoT data?

For most IoT workloads, Apache Flink is a strong default because it handles event time, large keyed state, and late data well. Kafka Streams suits smaller pipelines embedded in microservices, and managed Flink services remove cluster operations. The right choice depends on state size, team skills, and where the rest of your platform runs.

IoT streaming works when each layer does its own job

Reliable IoT streaming comes from a clear division of work. MQTT connects devices over unreliable networks, Kafka stores and distributes every reading, and a stream processor applies event time, state, and enrichment before results reach dashboards, alerts, and long-term storage. Designing for late data, schema changes, and reconnect storms from the start is what keeps the pipeline correct as the fleet grows.

Mimacom designs, builds, and runs IoT streaming pipelines on IBM Confluent, with production streaming delivery in manufacturing, transport, and retail. We have been a Confluent partner since 2017 and an official Confluent training partner since 2021, and we work with IBM Confluent as an IBM partner since IBM acquired Confluent. For a German heating manufacturer, we migrated an IoT data platform to cloud-native Kafka on Azure. Our Confluent consulting services cover architecture, implementation, and operations, and our Data Streaming Foundation for AI whitepaper explains how the same foundation supports analytics and AI.