Serverless stream processing means running real-time data pipelines without provisioning, sizing, or operating the servers or clusters underneath them. The platform scales compute up and down with the volume of incoming events, and you pay for what you use. Your team writes the processing logic, whether that is a function or a SQL statement, and the provider runs it.
The idea is attractive, because stream processing clusters are some of the hardest systems to size and keep healthy. But "serverless" covers two quite different approaches in streaming, and they are not interchangeable. Choosing the wrong one is the most common reason serverless streaming projects run into limits.
In general, a serverless service has three characteristics. There are no servers or clusters for you to provision. Capacity scales automatically with demand, sometimes down to zero. Billing is based on usage rather than on reserved capacity. Applied to stream processing, it means events are filtered, transformed, joined, or aggregated as they arrive, without your team managing the compute that does the work.
The first model is functions as a service. A function such as AWS Lambda is triggered by new records on a stream, processes them in small batches, and then stops. AWS's Well-Architected Serverless Lens describes this pattern with Amazon Kinesis as the stream, Lambda as the consumer, and DynamoDB as the store for results.
The second model is a serverless stream processing engine. Here a full stream processing framework, usually Apache Flink, runs as a managed service that scales itself. Confluent Cloud for Apache Flink is an example: you submit Flink SQL statements, and the service handles autoscaling, parallelism, and load balancing, including scaling to zero when a workload is idle. The engine keeps running continuously, which matters for anything that depends on state.
Most serverless streaming architectures share the same shape. Producers such as applications, devices, and databases publish events to an event streaming platform like Apache Kafka or Amazon Kinesis. A serverless processor reads from that platform, applies the logic, and writes results to a sink: another topic, a database, a data warehouse, or an application that reacts to them.
The difference between the two models sits in the processor. A function receives a batch of records, handles it, and ends. Any state it needs between batches has to be passed along or stored externally. A serverless engine runs as a long-lived job with its own managed state, so it can hold the context of a window or a join across millions of events without an external database.
The right choice depends on what the processing has to remember. This table compares the two serverless models with the self-managed option most teams know.
| Functions (e.g. AWS Lambda) | Serverless stream processing engine (e.g. serverless Flink) | Self-managed engine (Flink, Kafka Streams) | |
|---|---|---|---|
| Infrastructure | None to manage | None to manage | Clusters to size, patch, and monitor |
| State | Passed between invocations or stored externally | Managed by the engine | Managed by the engine, operated by you |
| Windowing | Short tumbling windows (up to 15 minutes on Lambda) | Tumbling, sliding, and session windows of any length | Tumbling, sliding, and session windows of any length |
| Scaling | Per batch, tied to shards or partitions | Automatic, within limits you set | Manual or scripted |
| Cost model | Per invocation and duration | Per unit of compute consumed | Reserved infrastructure plus staff time |
| Best fit | Stateless, spiky, event-by-event logic | Stateful pipelines without operations overhead | Very large, steady workloads with in-house expertise |
Stateless work treats each event on its own. Filtering out irrelevant records, routing events to the right destination, masking personal data, or enriching an event with a lookup are all stateless. These tasks fit functions well, particularly when traffic is spiky and you do not want capacity sitting idle between peaks.
Stateful work needs memory of earlier events. Counting transactions per customer per hour, joining an order stream with a shipment stream, or detecting a sequence of events that indicates fraud all depend on state that lives across many events. A serverless engine is designed for this. It checkpoints state, recovers it after a failure, and handles late events without you building that machinery yourself.
Functions can handle simple aggregations, but their limits show up quickly as requirements grow. Lambda's tumbling windows are capped at 15 minutes, and the state passed between invocations within a window is limited in size. Anything longer or larger has to go to an external store, which adds latency, cost, and a component to operate.
Delivery semantics need attention too. AWS notes that duplicate records can occur with Kinesis, so consumers and producers have to be idempotent. Concurrency is tied to the number of shards or partitions, which means throughput planning has not disappeared, it has moved.
Cost is the last consideration. Usage-based billing is efficient for bursty or low-volume streams. For a pipeline that processes high volumes around the clock, paying per invocation or per compute unit can cost more than capacity you reserve. The serverless engines address part of this by letting you cap the maximum compute a workload can use.
Each major cloud offers at least one way to process streams without managing clusters:
If your events already live in Kafka, a serverless engine that reads Kafka natively avoids copying data into a separate cloud stream first. If you prefer to build visually instead of writing code, no-code stream processing tools offer another route.
Start with the processing logic, not the platform. If each event can be handled on its own and traffic arrives in bursts, functions are usually the simplest and cheapest option. If the pipeline needs joins, windows longer than a few minutes, or exactly-once results, a serverless stream processing engine removes most of the operational work without forcing you into the limits of functions.
Then look at volume, skills, and existing systems. Steady, very high throughput can make self-managed or reserved capacity more economical, provided your team can operate it. Most organizations end up with a mix: functions for lightweight event handling, a serverless engine for stateful pipelines, and a clear rule for deciding which workload goes where.
Mimacom designs, builds, and runs data streaming platforms on Apache Kafka and Apache Flink. As an IBM partner with Confluent consulting services, we help teams move stream processing workloads to Confluent Cloud, including serverless Flink, and decide which logic belongs in functions, managed engines, or self-managed clusters.
Our teams have delivered production streaming solutions in banking, retail, transport, and manufacturing. That experience shapes how we approach state, delivery guarantees, and cost before a pipeline goes live, so that serverless reduces operational work instead of moving the problems somewhere else.
AWS Lambda works well for stateless stream processing, such as filtering, routing, and enriching records from Kinesis or Amazon MSK, especially when traffic is spiky. It is less suited to stateful work. Tumbling windows are limited to 15 minutes, and longer windows or joins need external storage or a dedicated stream processing engine.
Yes, if you use a serverless stream processing engine rather than functions. Services such as Confluent Cloud for Apache Flink manage state, checkpoints, and recovery for you while scaling automatically. Functions can only carry limited state between invocations, so they are a poor fit for long windows, joins, or pattern detection.
It depends on the traffic pattern. Usage-based pricing is usually cheaper for bursty, low, or unpredictable volumes, because you do not pay for idle capacity, and it removes the staff time spent operating clusters. For steady, very high throughput, reserved or self-managed capacity can cost less, so it is worth modeling both before you commit.
Serverless stream processing pays off when you match the model to the workload: functions for simple, bursty event handling and serverless engines for stateful pipelines. Talk to our Confluent team about which of your streaming workloads can run serverless, or explore our Confluent consulting services.