Compare LLM gateways, AI observability tools, and data control planes on enforcement point, data awareness, audit depth, and hybrid deployment.
By
Billy Allocca
Table of Contents
AI Governance Platforms Compared: Gateways vs Control Planes
AI governance tools fall into three groups: LLM gateways that control which models an application can call, observability tools that record what models and agents did, and data control planes that apply policy using data classification and user identity. This guide compares the three on enforcement point, data awareness, audit depth, cost attribution, and deployment options, including hybrid and on-prem estates.
In NexusOne's 2026 State of the Data Estate report, conducted by Enterprise Management Associates (EMA) in September 2026 (n=202) and commissioned by NexusOne, 78% of data and analytics decision-makers said they can enforce one policy definition fully or mostly everywhere, yet 81% said they must update two or more systems when one business rule changes [1]. That gap between stated control and the work behind it is the reason the category you choose matters. The architectural argument for putting governance in one layer is in the earlier post on the enterprise AI governance control plane; this guide is the practical comparison of the tools that claim the job.
What Does Governing AI at the Model Boundary Mean?
Governing AI at the model boundary means checking every request before it reaches a model and every response before it reaches a user or agent, then recording what happened and why. The model boundary is the point where a prompt, the data retrieved to answer it, and any tool calls cross from your systems into a model, whether that model runs on your own GPUs or behind a provider's API. A policy enforcement point is the component that applies a rule at that moment, allowing, blocking, redacting, or rerouting the request.
Five things cross the boundary, and a complete governance setup accounts for each one:
The prompt, which can carry personal data or a prompt injection, an instruction crafted to override the system's rules.
Retrieved data, the rows, documents, or vectors pulled in to ground the answer, which carry their own access rules.
Tool calls, the actions an agent proposes to take in other systems.
The output, which can leak sensitive information the user should never see.
The cost, measured in tokens, which needs an owner and a limit.
The OWASP Top 10 for LLM Applications names two of these risks directly: sensitive information disclosure (LLM02:2025) and unbounded consumption (LLM10:2025), for which it recommends rate limiting and per-user quotas [23][24]. Regulation adds the recording requirement. Under Article 26(6) of the EU AI Act, deployers of high-risk AI systems must keep automatically generated logs for at least six months unless other Union or national law says otherwise [22].
What Are the Three Categories of AI Governance Tools?
The three categories differ mainly in where they sit and what they can see. An LLM gateway is a proxy that every application calls instead of calling model providers directly, so it can issue keys, enforce budgets, and screen prompts. An AI observability tool collects traces, structured records of each step in a model or agent run, and scores them after the fact. A data control plane sits between users, agents, and the data estate itself, so it can evaluate policy using who is asking and what the data is.
A fourth group, AI governance and risk suites, sits beside all three. These manage the inventory of models and use cases, risk assessments, and compliance evidence against frameworks such as the NIST AI Risk Management Framework, first released in January 2023 and extended by the Generative AI Profile (NIST AI 600-1) on July 26, 2024 [20], and ISO/IEC 42001, the AI management system standard published in December 2023 [21].
Category | Where it sits | What it enforces or records | Sees data classification? | Named examples | Best fit |
|---|---|---|---|---|---|
LLM gateway | Between applications and model endpoints | Keys, budgets, rate limits, fallbacks, prompt and response guardrails | Only what is in the prompt text | LiteLLM, Portkey, Kong AI Gateway, TrueFoundry, Bifrost | Many apps calling many hosted models, with spend and key sprawl as the main problem |
AI observability | Alongside the application, receiving traces | Records prompts, retrieval steps, tool calls, cost, and quality scores | Detects sensitive data in recorded traces | Langfuse, Arize Phoenix, Maxim, Datadog LLM Observability | Debugging, evaluation, and quality monitoring of production AI |
AI governance and risk suite | Above the stack, as a system of record | Model inventory, risk assessments, policy packs, compliance evidence | Depends on integrations | Microsoft Purview with Microsoft Foundry, IBM watsonx.governance, Credo AI | Regulatory programs that need an AI register and documented controls |
Data control plane | Between users or agents and the data estate, at request entry | Identity-aware access, classification-based policy, PII screening, routing, cost per principal | Yes, from the catalog and tags | NexusOne | Agents and AI queries that reach governed data across several systems and environments |
Which Criteria Separate AI Governance Platforms?
Seven criteria separate the categories, and the first two decide most evaluations. Data awareness is whether the tool knows the classification of the data a request will touch, such as a column tagged as PII or a table restricted to one region. Identity federation is whether the tool carries the caller's corporate identity (from Active Directory, Okta, or another identity provider) to every system the request reaches, so a policy written for a person also binds an agent acting on that person's behalf.
Criterion | Question to ask a vendor | LLM gateway | AI observability | Data control plane |
|---|---|---|---|---|
Enforcement point | Where is the allow or block decision made? | At the gateway, before the model | None inline; analysis after the request | At request entry and at the data read |
Data awareness | Does policy use data classification and lineage? | Prompt text only | Scans recorded traces | Uses catalog tags and classifications |
Identity federation | Does the caller's identity reach the data systems? | Keys map to teams or users at the gateway | Records user IDs attached to traces | Same identity governs people, jobs, and agents |
PII screening | Can sensitive data be blocked or redacted inline? | Yes, through guardrail plugins | Detection and redaction in traces | Yes, at the edge and at the data layer |
Audit with reasoning | Does the log record why a request was allowed or blocked? | Request logs; check current documentation for reasoning | Full step-by-step traces | Allowed and blocked requests with the rule applied |
Cost attribution | Can spend be assigned to a team, user, or agent? | Yes, per key, team, or user | Cost per trace or user | Per role, user, or agent |
Hybrid deployment | Does it run on-prem, in your cloud, and air-gapped? | Several gateways self-host | Several tools self-host | Depends on the vendor |
Cost attribution means assigning every token of spend to the principal that caused it, the user, team, application, or agent, so budgets can be enforced at that level. Here the survey shows a gap worth testing: 92% of respondents said they can set AI spend limits at some level, but only 47% can set them at team, application, and agent level together (n=202) [1].
Which Vendors Fit Each Category?
Most vendors cover one category well and touch a second. The table lists only capabilities stated in each vendor's current documentation; where a capability is unconfirmed, the cell says so.
Vendor | Category | Verified capabilities | Deployment |
|---|---|---|---|
LiteLLM | Gateway | Virtual keys; hard budgets per key, team, org, and model; RPM and TPM limits; PII masking and prompt-injection guardrails through Presidio or Lakera; logging to Langfuse, Phoenix, and OpenTelemetry; SSO and audit logs in the Enterprise tier [2] | MIT-licensed; self-host anywhere, including air-gapped [2] |
Portkey | Gateway | Integrated guardrails; budget and rate limits on cost or tokens; fallbacks and conditional routing; simple and semantic caching [3] | Open-source gateway, self-hosted or hybrid [3][4] |
Kong AI Gateway | Gateway | AI PII Sanitizer on request or response bodies; AI Prompt Guard and AI Semantic Prompt Guard allow and deny rules [5]; AI Rate Limiting Advanced by tokens or by cost [6] | Check current documentation |
TrueFoundry AI Gateway | Gateway | Per-user, per-model, and per-application rate limits; budget limits with team cost attribution; guardrails for PII, prompt injection, and moderation; RBAC with scoped keys; MCP gateway [7] | SaaS, hybrid, or self-hosted in your VPC [7] |
Bifrost (Maxim) | Gateway | Hierarchical budgets through virtual keys, teams, and customers; rate limiting and access control [8] | Apache 2.0 open source [8] |
Langfuse | Observability | OpenTelemetry-based tracing of LLM and non-LLM calls; evaluations; prompt management; cost tracking per user [9] | Open source and self-hostable [9] |
Arize Phoenix | Observability | OpenTelemetry and OpenInference tracing; LLM, code, and human evaluations; datasets and experiments [10] | Elastic License 2.0, free to self-host [11]; Arize AX is the managed platform [10] |
Maxim | Observability | Simulation, evaluation, and observability of agents, with production log monitoring [12] | Check current documentation |
Datadog LLM Observability | Observability | Traces across prompts, retrieval, tool calls, and agent decisions; evaluators; flags prompt injection and PII exposure [13]; Sensitive Data Scanner redacts prompt-response pairs [14] | Check current documentation |
Microsoft Purview and Microsoft Foundry | Governance suite | Purview DSPM inventories AI apps and agents and audits prompts and responses [15]; Foundry guardrails detect prompt attacks and PII (preview) at user input, tool call, tool response, and output, and can annotate and block [16] | Microsoft cloud services |
IBM watsonx.governance | Governance suite | Model inventory, AI Factsheets, risk assessment questionnaires, fairness and drift monitors, external model tracking [18] | Check current documentation |
Credo AI | Governance suite | AI inventory of models, vendors, and agents; policy packs for the EU AI Act, NIST AI RMF, ISO 42001, and 20+ standards [19] | Check current documentation |
The overlap is growing. Maxim ships both an observability product and the Bifrost gateway, LiteLLM forwards logs to Langfuse and Phoenix, and TrueFoundry adds MCP tool governance to its gateway, so a single purchase can cover two categories for teams whose data sits in one place.
Why Can a Dashboard That Says Enforced Be Wrong?
A dashboard reports enforcement at the point it controls, but a policy can be bypassed or applied differently at every other point a request touches. A gateway that screens prompts has no record of the rows an agent later fetches through a tool, and a data platform that masks a column has no record of a copy of that column sitting in another system with its own rule.
The survey shows how often stated control and reported outcomes diverge. Respondents who say one policy can be enforced fully or mostly everywhere total 78% (27% fully, 51% mostly), yet 72% must implement at least some policies separately per system (n=202) [1]. Among respondents claiming full or mostly full enforcement (n=149), those whose rules live in two or more systems were more likely to report materially different AI decisions from the same information at least occasionally, 65% against 36% where the rule lives in one system [1]. The one-system group is small (n=22), so read this as an early association.
Spend shows the same pattern. Spend limits exist at some level for 92% of respondents, yet 57% said AI and inference costs ran above projection, 16% well above (n=202) [1]. Audit depth is thinner than it looks too: among respondents who can separate fixed rules from model inference for at least some decisions (n=194), only 43% could show exactly which rules and data produced a decision in all cases [1].
Vendor documentation names the same seams. Microsoft states that Purview data security policies for Foundry apply to interactions using Microsoft Entra ID user-context authentication against Foundry's managed inference endpoint, while calls made with other authentication methods appear in audit but are not enforced by those policies [17]. That is a sound, documented design, and it means "enforced" depends on how each caller authenticates. Test any tool by sending the same restricted request through every path you run, including the gateway, a direct SDK call, an agent tool, and a notebook, and compare the results.
When Is an LLM Gateway Enough?
An LLM gateway is the right answer when your main problems are model access, key sprawl, and spend, and the sensitive data question is already handled by the systems the prompts draw from. A gateway such as LiteLLM, Portkey, TrueFoundry, or Kong AI Gateway gives you one endpoint for many providers, budgets per team, fallbacks when a provider fails, and inline guardrails for PII and prompt injection [2][3][5][7]. Several are open source and can be self-hosted [2][4][8].
A gateway is usually enough when these conditions hold:
Applications call hosted models with prompts written by your own code, and retrieval comes from sources that already enforce access per user.
Cost control means budgets per key or team, and per-agent limits are a later need.
Audit needs are met by request and response logs, with traces sent to an observability tool.
Your data and models sit in one or two clouds, and nothing has to run air-gapped.
If all four hold, buy or deploy a gateway, add an observability tool for evaluation, and revisit the decision when agents start reading across systems. Paying for a data control plane in this situation buys capability you will not use yet.
When Do You Need Data-Aware Policy?
You need data-aware policy when the right answer to "may this request proceed?" depends on what the data is and who is asking, across more than one system. That happens as soon as agents retrieve from several stores, join them, and act. A gateway sees a prompt that asks for "all open claims in Ontario," but only a layer that knows the claims table is tagged as regulated personal data, and that the caller's role excludes one province, can apply the rule correctly.
Four signals mean you have crossed that line:
Agents act on data in several systems. Among respondents who reimplement policy per system (n=146), 52% reimplement agent identity separately in each, 38% of all respondents (n=202) [1].
Policy is tied to classification. Access permissions (67%), sensitive-data tagging (57%), and masking (53%) are the policies most often reimplemented per system among the same 146 respondents [1].
Audit must name the data behind a decision. Auditability of AI decisions is the second-weakest capability respondents reported, at 37% (n=202) [1].
The estate is hybrid. When data sits in an on-prem Hadoop cluster, a cloud warehouse, and object storage, a control that runs only in one cloud leaves the others unchecked.
The pattern for agents specifically is covered in the guide to AI agent access control, and the case for one governance model spanning data and AI is in one governance model for data and AI.
Where NexusOne Fits
NexusOne is a data and AI control plane: one place to set identity, policy, audit, and cost rules, with every system in the estate obeying them and no data moved. The NexusOne AI Data Control Plane sits where AI requests enter. It screens each request for jailbreaks and PII at the edge, logs allowed and blocked requests with the reasoning, and routes the request: deterministic questions go to query engines such as Trino at zero tokens, then to small models, and only then to frontier models. NexusOne calls this intelligent AI query routing. Spend is attributed to the person or agent that caused it, with per-role and per-agent budgets and circuit breakers, which are automatic stops when a budget is exceeded; the budget model is covered in AI token spend governance.
The data awareness comes from the layer underneath. Keycloak federates identity from Active Directory, Okta, LDAP, or SAML, and one identity and policy model in Apache Ranger governs humans, jobs, and agents across Trino, Apache Spark, Apache Kyuubi, and object storage. DataHub supplies classification tags and lineage, and the Apache Gravitino catalog exposes the Iceberg REST protocol and federates external catalogs. It runs identically on-prem, in any cloud, hybrid, or air-gapped on a Kubernetes cluster you control, using NexusOne's published 5-5-5-5 deployment model: about 5 minutes to provision, 5 hours to deploy, 5 days to the first governed workload, and 5 weeks to enterprise production, with Embedded Builders working alongside your team. A top-three US bank connected 30 applications in under 4 weeks after running three legacy Hadoop vendors, eliminating more than $100 million in license and hardware costs.
If your AI traffic is a few applications calling hosted models in one cloud, a gateway is the better fit and NexusOne is more than you need. Where a gateway is already in place, NexusOne can run alongside it and govern the data the gateway cannot see. See the NexusOne AI Data Control Plane, or book an expert consultation to map your request paths.
Key Takeaways
AI governance tools split into LLM gateways, observability tools, and data control planes, with governance and risk suites as the system of record beside them.
Enforcement point and data awareness decide most evaluations: gateways see prompts, observability tools see traces, and data control planes see identity and data classification.
In the EMA survey (n=202), 78% said one policy is enforced fully or mostly everywhere while 81% update two or more systems when one rule changes.
An LLM gateway is the right answer for model access, key management, and spend when retrieval already enforces per-user access.
Data-aware policy becomes necessary when agents act across several systems, policy follows classification, audits must name the data behind a decision, or the estate is hybrid.
Test "enforced" by sending one restricted request through every path you run and comparing the results.
Frequently Asked Questions
What Is the Best Platform for Governing AI Usage and Enforcing Policy at the Model Boundary?
The best platform depends on what your policy needs to know. If rules concern models, keys, and spend, an LLM gateway such as LiteLLM, Portkey, Kong AI Gateway, or TrueFoundry enforces them before each model call. If rules depend on data classification and the caller's identity across systems, choose a data control plane that evaluates policy at request entry and at the data read.
What's the Best Enterprise Platform for Governance and Audit of AI Queries Across Hybrid Data Estates?
Choose a platform that runs identically on-prem, in each cloud, and air-gapped, carries one identity to every data system, and logs allowed and blocked AI queries with the rule applied. Gateways and observability tools cover the model call and its trace. A data and AI control plane such as NexusOne also governs the data each query reads, wherever it sits.
What Is the Difference Between an LLM Gateway and an AI Governance Platform?
An LLM gateway is a proxy between applications and model providers that enforces keys, budgets, rate limits, and prompt guardrails. "AI governance platform" is a broader label covering gateways, observability tools, risk suites that keep the AI inventory and compliance evidence, and data control planes. Most enterprises run a gateway as one part of governance, alongside the systems that govern data access.
How Does NexusOne Compare to LiteLLM or Portkey?
LiteLLM and Portkey are LLM gateways focused on model access, budgets, routing, and prompt guardrails, and both offer open-source, self-hosted options. NexusOne is a data and AI control plane that adds identity-aware, classification-based policy across the data systems agents read from, plus routing that answers deterministic questions from query engines at zero tokens. For model access alone, a gateway is the simpler choice.
Which Standards Should an AI Governance Program Map To?
Most programs map controls to the NIST AI Risk Management Framework and its Generative AI Profile (NIST AI 600-1), ISO/IEC 42001 for the AI management system, and the OWASP Top 10 for LLM Applications for technical risks such as sensitive information disclosure. Organizations deploying high-risk systems in the EU also plan for the EU AI Act, including keeping automatically generated logs for at least six months.
References
More Tools, Less Control: The 2026 State of the Data Estate Report, NexusOne, published September 30, 2026 (survey conducted by Enterprise Management Associates (EMA), September 2026, n=202 data and analytics decision-makers at organizations with 500+ employees in the US, Canada, the UK, and France). https://nexus0ne.substack.com/p/more-tools-less-control-the-2026, retrieved October 20, 2026.
LiteLLM gateway product page and FAQ, LiteLLM (BerriAI). https://www.litellm.ai/, retrieved October 20, 2026.
AI Gateway, Portkey Documentation. https://portkey.ai/docs/product/ai-gateway, retrieved October 20, 2026.
Portkey AI Gateway (open-source repository), Portkey, GitHub. https://github.com/portkey-ai/gateway, retrieved October 20, 2026.
Kong AI Gateway Policies, Kong Docs. https://developer.konghq.com/ai-gateway/policies/, retrieved October 20, 2026.
AI Rate Limiting Advanced plugin, Kong Docs. https://developer.konghq.com/plugins/ai-rate-limiting-advanced, retrieved October 20, 2026.
AI Gateway, TrueFoundry Documentation. https://truefoundry.com/docs/ai-gateway, retrieved October 20, 2026.
Bifrost (open-source AI gateway repository), Maxim AI, GitHub. https://github.com/maximhq/bifrost, retrieved October 20, 2026.
Langfuse Documentation, Langfuse. https://langfuse.com/docs, retrieved October 20, 2026.
Phoenix: AI Observability and Evaluation, Arize AI Documentation. https://arize.com/docs/phoenix, retrieved October 20, 2026.
Phoenix self-hosting license, Arize AI Documentation. https://www.arize.com/docs/phoenix/self-hosting/license.md, retrieved October 20, 2026.
Maxim Documentation, Maxim AI. https://www.getmaxim.ai/docs, retrieved October 20, 2026.
LLM Observability, Datadog. https://datadoghq.com/product/llm-observability, retrieved October 20, 2026.
Sensitive Data Scanner evaluation, Datadog LLM Observability Documentation. https://docs.datadoghq.com/llm_observability/evaluations/sensitive_data_scanner, retrieved October 20, 2026.
Learn about Microsoft Purview Data Security Posture Management (DSPM), Microsoft Learn. https://learn.microsoft.com/en-us/purview/data-security-posture-management-learn-about, retrieved October 20, 2026.
Guardrails and controls overview in Microsoft Foundry, Microsoft Learn, updated October 14, 2026. https://learn.microsoft.com/en-us/azure/foundry/guardrails/guardrails-overview, retrieved October 20, 2026.
Manage compliance and security in Microsoft Foundry, Microsoft Learn. https://learn.microsoft.com/en-my/Azure/foundry/control-plane/how-to-manage-compliance-security, retrieved October 20, 2026.
Governing assets with watsonx.governance, IBM Documentation. https://www.ibm.com/docs/SSYOK8/wsj/analyze-data/xgov-overview.html, retrieved October 20, 2026.
AI Governance Platform, Credo AI. https://www.credo.ai/pl/ai-governance-platform-and-services, retrieved October 20, 2026.
Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1), National Institute of Standards and Technology, July 26, 2024. https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence, retrieved October 20, 2026.
ISO/IEC 42001:2023, Information technology, Artificial intelligence, Management system, International Organization for Standardization, December 2023. https://www.iso.org/standard/42001, retrieved October 20, 2026.
Article 26: Obligations of Deployers of High-Risk AI Systems, EU Artificial Intelligence Act (Regulation (EU) 2024/1689). https://artificialintelligenceact.eu/article/26/, retrieved October 20, 2026.
LLM02:2025 Sensitive Information Disclosure, OWASP Gen AI Security Project. https://genai.owasp.org/llmrisk/llm022025-sensitive-information-disclosure/, retrieved October 20, 2026.
LLM10:2025 Unbounded Consumption, OWASP Gen AI Security Project. https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/, retrieved October 20, 2026.
