A latency budget for every AI use case, the four-component pattern behind production real-time AI, and how to feed inference from a lakehouse without copying the data twice.
By
Billy Allocca
Table of Contents
Real-Time Data for AI Agents: The Streaming Architecture Patterns That Scale in Production
Real-time data for AI agents means joining a live event stream with precomputed historical context and delivering the result to an inference endpoint inside a fixed latency budget: under 100 milliseconds for fraud, a few seconds for agents. The join sets the floor, and the architecture that scales separates event stream, feature store, serving layer, and feedback loop.
Stripe's fraud system evaluates more than 1,000 characteristics of a transaction and returns a decision in under 100 milliseconds [1]. Mastercard's Decision Intelligence Pro scores a transaction in under 50 milliseconds after scanning roughly one trillion data points [2]. Neither figure is reachable by querying a warehouse at authorization time, because analytical engines are tuned for large scans and answer in seconds. Confluent's May 2025 Data Streaming Report, a survey of 4,175 IT leaders in 12 countries, found that 87 percent expect streaming platforms to be deployed more widely to feed AI systems with current data [3].
If you are still deciding what agents need from the data layer in general, the companion guide on what multi-agent systems need from your data layer covers identity and governance before this piece takes up latency.
What Is a Real-Time Data Architecture for AI Agents?
A real-time data architecture for AI is a join pipeline with a clock on it. Every low-latency prediction combines what just happened (a card swipe, a click, a tool call from an agent) with what the system already knows about the entity involved (this card's 30-day spend, this user's last 50 views). The first input arrives on an event stream; the second lives in historical storage. The prediction is only as fresh as the slower input and only as fast as the join between them.
Netflix described the resulting split in 2013 as three tiers: offline batch jobs with no latency constraint, nearline jobs that compute as events arrive and store results for later, and online components that must answer "while our member is waiting for recommendations to appear" and so carry a response-time SLA [4].
Two terms recur. An event stream is an append-only, ordered log of things that happened, read by many consumers at their own pace; Apache Kafka is the reference implementation and publishes latencies as low as 2 milliseconds [5]. A feature store is the system that computes model input values ("transactions in the last 10 minutes for this card"), keeps an offline copy for training and an online copy for serving, and generates point-in-time-correct training sets [6].
How Much Latency Can Each AI Use Case Tolerate?
Write down two numbers per use case before choosing any tool: the target response time and the tolerable staleness of the data behind it. They are independent. A fraud model must answer in under 100 milliseconds but can use features a few seconds behind reality; a nightly churn score can take hours but should see all of yesterday. Amazon's A/B tests, reported by Greg Linden in 2006, delayed page render in 100-millisecond increments and found that "even very small delays would result in substantial and costly drops in revenue" [7], which is why personalization budgets are set by page load and fraud budgets by the authorization window.
AI use case | Target response time | Tolerable data lag | Architecture that meets it |
|---|---|---|---|
Card fraud detection | Under 100 ms end to end [1]; under 50 ms for the score [2] | Seconds | Kafka stream, Flink aggregates, in-memory online store, model served next to it |
Recommendation ranking | 100 to 200 ms inside page render [7] | Minutes for candidates, seconds for session context | Offline candidates in the lakehouse, nearline feature updates, online lookup plus ranking model |
Content personalization | Under one page load, about 200 ms | Minutes | Nearline profile updates to an online store, batch refresh of segment features |
Agent tool call | 500 ms to 2 s per call; model token generation dominates the wait | Seconds to minutes, by data type | Federated query over governed sources with a semantic cache; deterministic requests routed straight to the engine |
Batch scoring | Minutes to hours | Hours to a day | Spark or Trino over Iceberg tables; no online store |
Because an agent's own reasoning takes seconds, its tool calls rarely need sub-10-millisecond lookups. They need predictable latency, current data, and the access policy the human user would get. The MCP specification tells clients to implement timeouts on tool calls and log tool usage for audit [8], so an unbounded warehouse scan behind a tool is a reliability defect as well as a latency problem.
What Are the Four Components of a Streaming Data Pipeline for AI?
Every publicly documented production real-time AI system uses the same four parts. Uber's Michelangelo, described in 2017, loaded batch features from Hive into Cassandra on a schedule, computed near-real-time aggregates from Kafka into the same Cassandra store, and logged those features back to HDFS so that "the same data and batch pipeline is used for both training and serving" [9]. The pattern does not depend on Hadoop: estates that took the Hadoop modernization path without a multi-year migration run the same four parts with Iceberg on object storage in place of Hive on HDFS.
Component | Job | Reference tools | Failure mode when missing |
|---|---|---|---|
Event stream | Capture every change as it happens for many independent consumers | Apache Kafka [5]; Debezium for change data capture from MySQL, PostgreSQL, Oracle, SQL Server, MongoDB transaction logs [10] | Features are a day stale |
Feature store | Compute features once; serve offline for training and online for inference | Feast (online stores include Redis, DynamoDB, Cassandra, PostgreSQL) [6][11]; Tecton | Training and serving code diverge; production accuracy falls below evaluation |
Serving layer | Answer key lookups in single-digit milliseconds next to the model | Redis (sub-microsecond processing, about 200 microseconds over 1 Gbit) [12]; DynamoDB (single-digit millisecond reads at any scale) [13] | The join runs on the analytical engine and blows the budget |
Feedback loop | Log served features and predictions, join to outcomes, feed retraining | Kafka topic of served features written to Iceberg; Google's Rules of ML, Rule 29 [14] | Training-serving skew is never measured |
Apache Flink usually sits between the stream and the feature store. Flink is "a framework and distributed processing engine for stateful computations over unbounded and bounded data streams," keeps state local for low processing latency, and checkpoints it asynchronously for exactly-once recovery [15]. Its SQL layer supplies the joins that matter: interval joins between two streams inside a time window, temporal joins against a versioned table as of the event's timestamp, and lookup joins that enrich a row from an external system [16]. A temporal join gives a fraud model the card's limit as it was at the moment of the transaction, which a warehouse cannot do at streaming speed. The schema of each hand-off (topic, feature view, served-feature log) is what a data contract pins down; the companion guide on data contracts for AI agents covers how.
How Do You Bridge a Lakehouse to Real-Time Serving Without Duplicating Data?
The common failure is a second copy of the estate: Kafka, Flink jobs, and a key-value store that, a year later, disagree with the lakehouse about a customer's balance. An open table format such as Apache Iceberg prevents this by making the lakehouse table the shared substrate. Flink writes to it, Trino and Spark read from it, and the online store is a derived cache with a TTL, never a second system of record.
Pattern | Mechanism | Copies of data | Freshness bound |
|---|---|---|---|
Streaming write to Iceberg | Flink commits on each successful checkpoint; uncommitted files stay invisible; upsert mode uses equality fields on v2 tables [17] | One governed table | Checkpoint interval, seconds to a minute |
CDC mirror of operational systems | Debezium reads the transaction log, publishes to Kafka, a sink writes to Iceberg [10] | One mirror replacing ad hoc extracts | Seconds behind the source commit |
Federated read at query time | Trino's Iceberg connector queries the files Flink wrote, with snapshot isolation and time travel [18] | Zero additional copies | Last committed snapshot |
Materialized online features | The feature store pushes values from the offline store to Redis or DynamoDB instead of pulling on demand, "which reduces the latency of feature retrieval" [6] | A bounded, expiring cache | Materialization schedule plus streaming updates |
Served-feature log | Each inference appends inputs and output to a Kafka topic that lands in Iceberg | One table that is training set and audit trail | Same as the streaming write |
The online store is the only step that deliberately holds a second representation, and it should be treated like a CPU cache: small, expiring, rebuildable from the table of record. Iceberg's Flink documentation warns that expiring snapshots or deleting orphan files without preserving the Flink job's last snapshot can corrupt the job's state [17], so compaction has to be scheduled around the streaming writer.
Why Not Just Query the Warehouse Faster?
The warehouse is the wrong engine for point lookups under load, and tuning does not change its design. Analytical engines read columnar files in large blocks and plan for scans and aggregations, amortizing cost across seconds of work. A fraud decision needs a handful of values for one key, thousands of times per second, in the tail of the distribution; Tecton's serving SLO treats an online store read above 25 milliseconds as out of bounds [19]. A cloud-era warehouse is also billed by compute time, so routing authorization traffic to it turns a latency problem into a per-transaction cost problem.
Warehouses and federated engines still belong in the design as the offline and analytical tiers. Trino over Iceberg is the right engine for an agent asking "compare this quarter's churn across regions," where a two-second answer is acceptable and correctness across the whole estate outweighs a 5-millisecond lookup. The design question is which requests belong to which tier, and a router should decide that, never each agent's developer.
NexusOne Routes Analytical and Low-Latency Access Through One Query Layer
NexusOne is the query layer between the streaming stack and the lakehouse that decides where each request goes. Its semantic router classifies every request: a deterministic lookup or SQL question goes straight to the right engine through the federation layer, with no model in the path and zero tokens spent, while requests that need a model go to an existing model, an on-prem SLM, or a frontier model with PII redacted before egress. Agent and MCP tool calls ride the same router, governed by one identity model in Keycloak and one policy set in Apache Ranger, enforced across Trino, Apache Spark, Apache Kyuubi, and S3 object storage per object.
Streaming is a task the platform brings tools to. Pick "mirror a transaction system" and NexusOne provisions the change data capture path into Iceberg with one click; pick "stream data" and the pipeline lands in the same catalog, so a dataset tagged in DataHub carries the same Ranger policies on streaming and batch tables. The layer runs identically on-prem, in any cloud, or air-gapped, and teams on the 90-day path to an AI-ready data estate typically add streaming in month two. To pressure-test your latency budgets, talk with a NexusOne architect.
Key Takeaways
Real-time AI is a join between a live event and precomputed context, and the join, not the stream, sets the latency floor.
Fix two numbers per use case first: target response time and tolerable data lag; fraud needs under 100 ms with seconds of lag, agents tolerate seconds of response with minutes of lag.
The production pattern has four parts: event stream (Kafka, Debezium), feature store (Feast, Tecton), serving layer (Redis, DynamoDB), and a feedback loop that logs served features for retraining.
Apache Flink supplies the temporal and lookup joins that enrich events at streaming speed and commits results to Iceberg on each checkpoint.
Keep one table of record in Iceberg and treat the online store as an expiring cache.
A router that classifies requests by access pattern, under one identity and policy model, is what lets agents use both tiers safely.
FAQ
What Does a Real-Time Data Architecture for AI Agents Look Like?
It has four components: an event stream such as Apache Kafka, a feature store that computes and serves precomputed context, a serving layer such as Redis or DynamoDB colocated with the model, and a feedback loop that logs every served feature for retraining. A query router in front of these tiers sends deterministic lookups to the fastest path and analytical questions to a federated engine, under the identity and policy that govern human users.
How Is Streaming Data Used for AI in the Enterprise?
Stream processors like Apache Flink compute windowed aggregates (spend in the last 10 minutes, clicks in the current session) and push them to an online store so a model reads them in milliseconds at inference time. The same stream is written to Apache Iceberg, where it becomes the training set and the audit trail. Change data capture with Debezium extends this to operational databases by publishing their transaction logs as events.
How Does Apache Kafka Fit Into Enterprise AI Pipelines?
Kafka is the durable, ordered event log that decouples producers from every consumer. In an AI pipeline it carries raw events to Flink for feature computation, carries Debezium change events out of operational databases, and carries the served-feature log back to storage for retraining. With published latencies as low as 2 milliseconds, Kafka is rarely the bottleneck; the join and the online lookup are.
How Do You Get Low-Latency Data to an AI Inference Endpoint?
Precompute the expensive part and cache it next to the model. History-dependent features are computed offline or by a stream processor and pushed into a single-digit-millisecond key-value store, so the endpoint performs one key lookup instead of a query. Stripe's fraud decision, more than 1,000 characteristics in under 100 milliseconds, follows this shape. The online store must stay rebuildable from the lakehouse table of record so it never becomes a second source of truth.
References
How We Built It: Stripe Radar. Stripe Engineering. https://stripe.dev/blog/how-we-built-it-stripe-radar
Mastercard Supercharges Consumer Protection With Gen AI. Mastercard Newsroom, February 1, 2024. https://www.mastercard.com/us/en/news-and-trends/press/2024/february/mastercard-supercharges-consumer-protection-with-gen-ai.html
Data Streaming Enables AI Product Innovation, Say 90% of IT Leaders in Confluent's New Data Streaming Report. Confluent, May 20, 2025. https://www.confluent.io/press-release/data-streaming-report-2025/
System Architectures for Personalization and Recommendation. Netflix Technology Blog, March 27, 2013. https://netflixtechblog.com/system-architectures-for-personalization-and-recommendation-e081aa94b5d8
Apache Kafka. Apache Software Foundation. https://kafka.apache.org/
Introduction to Feast. Feast documentation. https://docs.feast.dev/
Marissa Mayer at Web 2.0. Greg Linden, Geeking with Greg, November 9, 2006. https://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html
Tools. Model Context Protocol Specification, 2025-06-18. https://modelcontextprotocol.io/specification/2025-06-18/server/tools
Meet Michelangelo: Uber's Machine Learning Platform. Uber Engineering Blog, September 5, 2017. https://www.uber.com/us/en/blog/michelangelo-machine-learning-platform/
Debezium Architecture. Debezium documentation. https://debezium.io/documentation/reference/stable/architecture.html
Online Stores. Feast documentation. https://docs.feast.dev/reference/online-stores
Diagnosing Latency Issues. Redis documentation. https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/latency/
Amazon DynamoDB. Amazon Web Services. https://aws.amazon.com/dynamodb/
Rules of Machine Learning: Best Practices for ML Engineering. Google for Developers. https://developers.google.com/machine-learning/guides/rules-of-ml
What Is Apache Flink? Architecture. Apache Flink. https://flink.apache.org/what-is-flink/flink-architecture/
Joins. Apache Flink SQL documentation. https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/sql/queries/joins/
Flink Writes. Apache Iceberg documentation. https://iceberg.apache.org/docs/latest/flink-writes/
Iceberg Connector. Trino documentation. https://trino.io/docs/current/connector/iceberg.html
Online Serving Monitoring. Tecton documentation. https://docs.tecton.ai/docs/monitoring/online-serving
