Most organizations running AI initiatives today are not short on data. They are short on a place to put it that works for analytics, machine learning, and generative AI at the same time without three separate copies and three separate pipelines. IBM watsonx.data is IBM's answer to that problem: an open lakehouse built to hold structured, semi-structured, and unstructured data in one place, queryable by multiple engines, and governed well enough that AI teams and analytics teams can work from the same source without stepping on each other. For teams already running Kafka for real-time data, the more interesting question is how that streaming data actually lands in a lakehouse built this way.
IBM watsonx.data is an open architecture lakehouse that separates compute, metadata, and storage into independent layers, so different query engines can access the same underlying data without duplicating it. It stores data using Apache Iceberg as its primary open table format, alongside support for Avro, Parquet, and ORC, and exposes that data to multiple query engines, including Presto (both the Java and C++ variants) and Spark. Governance runs through IBM Knowledge Catalog, which enforces schema consistency and manages metadata across the platform. It also ships with Milvus, a vector database, for embedding storage and similarity search, which is what connects the platform to AI and generative AI workloads rather than just traditional analytics.
The platform also supports no-copy access to data that already sits in existing systems, including Db2 Warehouse and Netezza Performance Server, which means a migration does not have to happen before a team can start querying that data through watsonx.data. Deployment options cover AWS, Azure, IBM Cloud, and on-premises through Red Hat OpenShift, and a Developer Edition packages pre-configured engines for local installation on macOS, Windows, or Linux, which is typically how teams evaluate the platform before committing to a production deployment.
A data lakehouse is an architecture that combines the low-cost, flexible storage of a data lake with the data management features of a data warehouse: ACID transactions, schema enforcement, and query performance. The idea is to avoid choosing between "cheap and flexible" and "structured and fast." Data lands once, in an open format, and both exploratory analytics and governed reporting run against the same copy.
| Aspect | Data warehouse | Data lake | Data lakehouse |
|---|---|---|---|
| Data structure | Highly structured, schema-on-write | Raw, structured or unstructured | Structured and unstructured, open table formats |
| Cost | Higher storage cost | Low-cost object storage | Low-cost object storage |
| Governance | Strong, built in | Often weak without added tooling | Built in through metadata and schema enforcement |
| Flexibility for diverse data | Limited | High | High |
| Common failure mode | Expensive to scale for raw data | Becomes an ungoverned data swamp | Requires deliberate table format and governance choices |
| Best suited for | SQL-based reporting | Raw data storage, exploration | Unified analytics, ML, and AI workloads |
The pattern that led here is familiar to most data teams. A warehouse gets built first for reporting, because reporting needs structure and speed. As raw and semi-structured data accumulates, usually faster than the warehouse can absorb it, a lake gets added alongside it for cheap storage and flexibility. Within a few years, the organization is running two systems, paying to keep them roughly in sync, and asking data scientists to pull from whichever one happens to have the data they need. The lakehouse is an attempt to stop paying that tax by making one system do both jobs well.
AI and machine learning workloads need large volumes of raw and semi-structured data for training, and they need it without a lengthy extract-transform-load cycle into a separate structured store. A lakehouse keeps that data in an open format that both a data scientist's Spark job and a business analyst's SQL query can read directly. In watsonx.data specifically, the addition of Milvus means embeddings and vector search sit alongside the same governed data, rather than in a disconnected vector store that has to be synchronized separately.
This matters most for retrieval-augmented generation, where a model's output is only as good as the context it retrieves. If the vector store holding embeddings is a separate system from the lakehouse holding the structured records those embeddings describe, keeping the two consistent becomes its own ongoing engineering problem. Housing both in watsonx.data removes that synchronization step, since the vector search and the underlying governed data are queried from the same platform rather than reconciled after the fact.
watsonx.data itself is a query and storage layer, not a stream processor, so Kafka topics do not land in it directly. In practice, streaming data reaches the lakehouse through an ingestion layer, typically a change-data-capture or stream-processing job, that consumes Kafka events and writes them into Iceberg tables as they arrive. Once that data is in table form, it looks like any other dataset in watsonx.data: queryable by Presto or Spark, governed by IBM Knowledge Catalog, and available to the same analytics and AI workloads as batch-loaded data. The practical benefit is that a team already running Kafka for real-time events does not need a separate real-time serving layer alongside its analytical store. The streaming pipeline feeds one governed table, and every downstream consumer, a dashboard, a batch report, or a model training job, reads from that same source instead of a separate real-time-only copy.
This does introduce a tradeoff worth naming honestly: the data is only as fresh as the ingestion job feeding it, not instantaneously current the way a native stream processor would be. For most analytics and AI use cases, table freshness measured in seconds to minutes is more than sufficient, and it is a significant improvement over a nightly batch load. For use cases that genuinely need sub-second processing, such as fraud detection acting on individual transactions, that logic still belongs in the stream-processing layer itself, not in the lakehouse.
Teams adopt watsonx.data most often for three kinds of problems: consolidating analytics across systems that have accumulated separate data marts over time, building a single governed source for both BI reporting and machine learning training data, and supporting retrieval-augmented generation workloads where Milvus-backed vector search needs to sit next to the structured data it is retrieving context from. A fourth, increasingly common case is near-real-time analytics, where Kafka-fed Iceberg tables let dashboards and models work against data that is minutes old rather than a nightly batch load.
A fifth case shows up specifically in regulated industries: using no-copy access to query data still living in an existing Db2 Warehouse or Netezza system, so a governance and access-control layer can be introduced without first completing a full data migration. This lets compliance and audit requirements get addressed on their own timeline, separate from the larger and slower project of consolidating storage.
The word "open" in an open lakehouse is not a marketing detail. It determines what happens to the data five years from now, not just today.
Because watsonx.data stores data in Iceberg, Parquet, Avro, and ORC rather than a proprietary format, the underlying data remains readable by any engine that supports those formats, not only IBM's. Switching or adding a query engine later does not require re-exporting or reformatting the data that already exists.
Open formats mean the same tables can be queried by Presto, Spark, or other Iceberg-compatible engines a team may already run elsewhere, without duplicating the data for each tool. This matters most in organizations that have accumulated multiple analytics and data science tools over time and cannot realistically standardize on a single engine.
Data outlives the tools built around it. An open table format means that as query engines, AI frameworks, and governance tooling change over the next several years, the underlying data does not need to be migrated or rewritten to keep up. The investment is in the data layer, not in any single vendor's engine.
Adopting watsonx.data is not a drop-in replacement for an existing warehouse or lake. It requires deciding which data sources move first, typically the ones already causing the most duplication or governance pain, and building the ingestion paths, batch or streaming, that will keep Iceberg tables current. Teams also need to decide how existing BI tools and machine learning pipelines will connect to the new query engines, since that often means updating connection strings and access policies rather than rewriting the tools themselves. None of this is a rip-and-replace project. It is closer to consolidating access to data that already exists, one source at a time.
Three decisions tend to determine whether the rollout goes smoothly. First, which query engine, Presto or Spark, handles which workload, since routing every query through a single engine regardless of shape tends to produce worse performance than matching the engine to the job. Second, how IBM Knowledge Catalog's governance policies map onto the access controls already enforced in source systems, so that consolidating access does not accidentally loosen it. Third, who owns the ingestion jobs feeding Iceberg tables from streaming or batch sources, since an ingestion pipeline with no clear owner is usually the first thing to break silently when a source schema changes upstream.
Mimacom designs lakehouse and streaming architectures for clients who need governed analytics and AI workloads running against the same data, not separate copies maintained by separate teams. That includes the ingestion layer connecting Kafka or other event streams into Iceberg tables, the governance model that keeps a lakehouse from becoming an ungoverned data swamp, and the migration path from an existing warehouse or lake into an open architecture like watsonx.data.
The platform decision matters less than most teams assume at the outset. What actually determines whether a lakehouse project succeeds is whether the ingestion paths, batch and streaming, reliably keep the underlying tables current and governed. Get that right, and the choice of query engine or AI framework on top becomes a much smaller decision to reverse later.
Not necessarily, and not immediately. Watsonx.data can query data in place in systems like Db2 Warehouse and Netezza without requiring migration, which lets teams consolidate access before deciding whether to fully retire an existing warehouse.
Kafka topics are not read directly by watsonx.data's query engines. A streaming or change-data-capture layer consumes the Kafka events and writes them into Iceberg tables, and watsonx.data's Presto and Spark engines then query those tables like any other dataset in the lakehouse.
Because the format determines what happens to the data if the platform changes. Iceberg, Parquet, Avro, and ORC are readable by many engines beyond IBM's own, so the data stays usable even if the query engine or vendor relationship changes later. The platform is replaceable. A proprietary data format is not.
Whether you are consolidating existing data marts or building a governed path from Kafka streams to AI workloads, Mimacom can design and implement the lakehouse architecture that fits your existing systems.