Streamhouse Architecture: How to Build One on IBM Confluent

Streamhouse Architecture: How to Build One on IBM Confluent

A streamhouse architecture captures, transports, transforms, governs, and serves the current state of a business continuously, so that production applications and AI agents can act on it. The open definition published in September 2026 sets out what a streamhouse must do, but it leaves the build to each organization. This guide covers that part: the six building blocks of a streamhouse, how they map to the Connect, Stream, Process, Govern, and Serve stages of IBM Confluent, and the order in which to put them together.

If you are new to the concept, start with our introduction to the streamhouse and its three principles: real-time, production-native, and decentralized. This article assumes you know what a streamhouse is and want to build one.

Streamhouse architecture at a glance

The Streamhouse Working Group names six building blocks. Each one has a clear job, and on the Confluent data streaming platform, now part of IBM, each one has a matching component.

Building blockJob in the architectureStageIBM Confluent component
Change data captureCaptures every insert, update, and delete in operational databases as it happensConnectFully managed CDC source connectors
Event streamsStores and transports business events durably, in order, for every consumerStreamApache Kafka
Stream processingJoins, filters, and enriches events into current views of the businessProcessApache Flink and Flink SQL
CatalogsRecords what data exists, what it means, who owns it, and where it flowsGovernStream Governance: Schema Registry, Stream Catalog, Stream Lineage
Open tablesShares streams with lakehouses and analytics engines as tablesServeTableflow, for Apache Iceberg and Delta Lake
Low-latency servingAnswers lookups from applications and AI agents in real timeServeReal-Time Context Engine, APIs, and SQL

The stages run in this order for each new use case, but the architecture is not a pipeline you build once. Each new source, view, or consumer reuses what is already in place, which is how a streamhouse grows across the business without a central consolidation project.

Step 1: Start from one decision that needs current data

The most common mistake is to start with the platform. A streamhouse built to stream everything has no first customer and no measure of success. Start instead with one decision that has to happen at machine speed: approving a card payment, rerouting a shipment, confirming stock for an online order, or flagging a claim for review.

For that decision, write down the facts it depends on, the systems that hold those facts, and how fresh each one needs to be. A payment approval might need the customer’s recent transactions from the core banking system, the device data from the mobile app, and the current fraud rules. That list becomes your scope for the next five steps.

Choosing one decision also tells you who consumes the data. If it is an AI agent, you will need low-latency serving. If it is a fraud model retrained weekly, you will also need open tables. Most first use cases need both.

Step 2: Connect sources with change data capture

Change data capture (CDC) reads a database’s transaction log and publishes every change as an event. It is the least invasive way to get current data out of operational systems, because the source application does not change and the database does not take extra query load.

Change data capture from operational databases

Confluent Cloud offers fully managed CDC source connectors for databases such as Oracle and PostgreSQL. They are built on Debezium and the Kafka Connect framework, and they write each row-level change to a Kafka topic. Because the connectors are managed, your team configures them rather than running and patching connector infrastructure.

Plan for the initial snapshot. When a CDC connector starts, it usually reads the existing table contents before switching to live changes, which can take hours for large tables. Schedule it, and decide whether consumers need the full history or only changes from a given point.

Connectors for applications, mainframes, and message queues

Not every source is a database you can read from. SaaS applications such as Salesforce are reached through connectors that use their APIs. Mainframe data from IBM Z typically arrives through IBM MQ, which Confluent connects to directly, or through a replication tool that publishes changes to Kafka. For each source, choose the option that captures changes as they happen. A connector that polls once an hour turns a streamhouse back into a batch system.

Step 3: Stream events on Apache Kafka

Every captured change lands in Apache Kafka, the event streaming layer of the streamhouse. Kafka stores events durably and in order, and any number of consumers can read the same stream independently. For a primer on topics, partitions, and consumers, see our guide to Apache Kafka for real-time data streaming.

Two design choices matter most here. The first is ownership. The decentralized principle means each domain owns its streams: the payments team owns payment events, the logistics team owns shipment events. Treat each important topic as a data product with a named owner, a documented schema, and an agreed level of service.

The second is retention. Some topics hold a history of events, such as every payment ever made. Others hold the current state, such as the latest status of each order. For current state, Kafka’s log compaction keeps the most recent value for each key and discards older ones, which keeps the topic small while still giving any new consumer the full current picture.

Step 4: Process data in motion with Apache Flink

Raw change events are rarely what an application or agent needs. A payment approval needs the transaction joined with the customer’s profile and their activity in the last ten minutes. Stream processing with Apache Flink builds those views continuously, as each event arrives, instead of in a nightly job.

On Confluent Cloud, Flink runs as a managed service, and most transformations can be written in Flink SQL. Typical jobs filter out noise, join streams from different systems, enrich events with reference data, and aggregate activity over time windows. Each job writes its result to a new Kafka topic, which becomes another governed stream that other teams can reuse.

Start with the view your first decision needs, and keep each job small enough that its owner can explain it. Our article on stream processing covers the underlying concepts, including state, time, and windows.

Step 5: Govern data in motion

Governance is where a streamhouse earns trust. Once applications and agents act on streams automatically, a malformed event or an exposed field becomes a production incident. Govern the data as it moves, rather than checking it after it lands somewhere. Confluent’s Stream Governance suite covers three areas.

Schemas and data contracts

Schema Registry stores the schema for each topic, in Avro, JSON Schema, or Protobuf, and checks that producers write data that matches it. Data contracts extend schemas with data quality rules and an agreed way to evolve, so a producer cannot break its consumers by renaming a field or sending an empty value where a valid one is required.

Discovery and lineage

Stream Catalog lets teams classify, tag, and find the streams that already exist, so a new project reuses the payments stream instead of building a second one. Stream Lineage shows how data flows from each source through processing jobs to its consumers, which tells you what will be affected before you change anything.

Protecting personal data in flight

Streams often carry personal or regulated data. Client-side field-level encryption encrypts sensitive fields before they leave the producer, so only authorized consumers can read them. Combine it with access controls on each topic so agents only see the data they are allowed to act on. Our guide to stream governance explains these controls in more depth.

Step 6: Serve current data to every consumer

The final stage delivers the current state of the business to the systems that use it. A streamhouse serves two kinds of consumer: analytics engines that read tables, and applications and AI agents that need answers in milliseconds.

Kafka to Iceberg with Tableflow

Tableflow materializes Kafka topics as Apache Iceberg or Delta Lake tables, without a separate batch pipeline. It publishes those tables to catalogs, including its own Iceberg REST catalog, AWS Glue, Databricks Unity Catalog, Snowflake Open Catalog, and Apache Polaris, so query engines can find and read them. Lakehouses that read Iceberg, such as IBM watsonx.data, can then query tables that stay current as events arrive. This is how a streamhouse keeps the lakehouse fresh rather than replacing it, as our comparison of the streamhouse vs. lakehouse explains.

Low-latency serving for applications and AI agents

Applications can consume Kafka topics directly or query views through APIs and SQL. For AI agents, Confluent’s Real-Time Context Engine materializes topic data into tables optimized for fast lookups and exposes them through the Model Context Protocol (MCP). Agents query current data through MCP tools without connecting to Kafka themselves. Our article on real-time data for AI agents explains why this matters for agentic AI in production.

Run it to production standards

The production-native principle means the streamhouse runs to the same service levels as the systems that depend on it. Once your first decision runs on live data, an outage in the data layer stops that decision. Before going live, check that you have:

  • A deployment model that fits your constraints: Confluent Cloud as a managed service, or Confluent Platform on your own infrastructure, compared in our guide to Confluent Cloud vs. self-hosted Kafka
  • High availability and disaster recovery: multi-zone clusters and a tested plan for failing over to another region, for example with Cluster Linking
  • Monitoring of data freshness as well as uptime: alerts on consumer lag and connector failures, since a running pipeline can still deliver stale data
  • Clear ownership: a named owner and an agreed level of service for each source, stream, processing job, and serving endpoint
  • Security reviews: access controls, encryption, and audit logging approved before agents receive write access to any system

FAQs

What are the components of a streamhouse architecture?

A streamhouse architecture has six building blocks: change data capture, event streams, stream processing, open table formats, catalogs, and low-latency serving. On IBM Confluent, these map to CDC connectors, Apache Kafka, Apache Flink, Tableflow, Stream Governance, and the Real-Time Context Engine, grouped into the Connect, Stream, Process, Govern, and Serve stages.

Do you need Confluent Cloud to build a streamhouse?

No. Streamhouse is a vendor-neutral architecture, and you can build one with open source components, commercial platforms, or a mix. On IBM Confluent, you can run Confluent Cloud as a fully managed service or Confluent Platform on your own infrastructure. Some managed features, such as Tableflow and the Real-Time Context Engine, are Confluent Cloud services, so check availability for your deployment model.

How do you stream Kafka data into Apache Iceberg?

On Confluent Cloud, you enable Tableflow on a Kafka topic, and it materializes the topic as an Apache Iceberg table and keeps it current as new events arrive. Tableflow registers the table in a catalog such as its Iceberg REST catalog or AWS Glue, and engines that read Iceberg can query it. Without Tableflow, the usual approach is a sink connector or a Flink job that writes to Iceberg.

How Mimacom can help

Mimacom is an IBM Silver Business Partner with a long-standing Confluent and Apache Kafka practice. Our teams have run streaming platforms in production for banks, retailers, manufacturers, and transport operators. We help clients choose the first decision to put on live data, build the connectors, streams, processing, and governance it needs, and hand over a platform their own teams can extend.

Build your streamhouse one decision at a time

Mimacom’s streaming architects help you plan your first streamhouse use case on IBM Confluent and build it to production standards. Explore our Confluent consulting services or contact us to get started.