Model routing sends each AI request to the cheapest model, or no model, that meets quality, latency, and policy limits. Learn the tiers, signals, and tools.
By
Billy Allocca
Table of Contents
What Is Model Routing? Matching Each AI Request to a Model
Model routing sends each AI request to the cheapest model, or no model at all, that can answer it within your quality, latency, and policy limits. Enterprises route across three tiers: a query engine for deterministic questions, a small specialized model for bounded tasks, and a frontier model for open-ended reasoning. The decision weighs task complexity, data sensitivity, the caller's role, budget, and latency.
Cost pressure makes routing urgent. In NexusOne's 2026 State of the Data Estate report, conducted by Enterprise Management Associates (EMA) in September 2026 (n=202) and commissioned by NexusOne, 57% of data and analytics decision-makers said their AI and inference costs ran above projection, and 16% said well above [1]. Asked how they responded to rising AI cost, 47% had capped usage, 36% had moved workloads to owned infrastructure, and 31% had substituted cheaper models (n=202) [1]. The survey did not ask about routing, so it describes the pressure rather than the fix. For a ranked view of the products in this category, see the companion guide to AI query routing in 2026.
What Is Model Routing?
Model routing is a decision step that sits between an application and its models and picks, for each request, which model (or which non-model system) should answer. A router is the component that makes that choice, usually with a lightweight classifier, a set of rules, or both. The application calls one endpoint, and the router forwards the request to the chosen target and returns the response, often with a field that says which model handled it [20].
The economic case rests on price spread. At list prices retrieved October 20, 2026, the most expensive and least expensive text models from the same provider differ by up to 100 times on output tokens [2][3]. Researchers measured the same spread earlier: the FrugalGPT paper found that fees for popular LLM APIs differ by two orders of magnitude [5].
Provider | Highest-priced model listed (input / output per 1M tokens) | Lowest-priced text model listed (input / output per 1M tokens) | Output price spread |
|---|---|---|---|
Anthropic | Fable 5.1: $10 / $50 [2] | Haiku 5.5: $0.10 / $0.50 for prompts up to 100K tokens [2] | 100 times |
OpenAI | gpt-6-astra: $10.00 / $50.00 [3] | gpt-6-luna: $0.10 / $0.50 [3] | 100 times |
gemini-4-argon: $2.00 / $10.00 through November 30, 2026, then $4.00 / $20.00 [4] | gemini-3.1-flash-lite: $0.25 / $1.50 [4] | About 7 times now, about 13 times from December 1 |
Prices change often, so treat this table as a snapshot and recheck the provider pages before you model a budget. Batch processing, which Anthropic discounts by 50% [2], and prompt caching shift the numbers further.
What Are the Three Routing Tiers, and What Goes to Each?
Most enterprise routing designs use three tiers, and the cheapest tier answers the requests that need no model at all. A deterministic question has one correct answer that already exists in a governed system, such as "What was Q3 revenue for the Northeast region?" A query engine answers it with SQL against the source, so it consumes zero tokens and returns the same number every time.
A small language model (SLM) is a compact model, often with a few billion parameters and often fine-tuned on one domain, that is cheap enough to run on your own GPUs. A frontier model is a provider's largest, most capable general model, priced at the top of the table above.
Tier | What answers | Typical requests | Token cost | Where it runs | Main failure mode |
|---|---|---|---|---|---|
1. Deterministic | Query engine or rules service over governed data | Metric lookups, record retrieval, status checks, policy rules | Zero tokens | Inside your estate | Wrong answer when the semantic definition is wrong or missing |
2. Small model | SLM or a lower-priced hosted model | Classification, extraction, summarization, routine drafting, domain Q&A | Lowest model tier | Self-hosted GPUs or a low-cost endpoint | Quality drops on multi-step reasoning or unfamiliar inputs |
3. Frontier model | Largest hosted or licensed model | Open-ended analysis, long multi-step reasoning, complex code, ambiguous requests | Highest tier, up to 100 times the small-model output price [2][3] | Usually a provider's cloud | Cost overruns and sensitive data leaving the estate |
Tier 1 depends on the semantic layer more than on the router. If "revenue" means three different things across three systems, the query engine returns a confident wrong number, so routing to it only works when the business definitions are governed. The NexusOne guide to why AI query routing is infrastructure covers that dependency in depth.
What Signals Drive a Routing Decision?
A router decides on five signals: task complexity, data sensitivity, the caller's role, the remaining budget, and the latency target. Complexity is the signal most products score automatically, and the other four are usually set by policy.
Signal | What it measures | How routers read it | Example rule |
|---|---|---|---|
Task complexity | How much reasoning the request needs | Classifier score, keyword rules, or predicted quality per model [11][17] | Send SIMPLE and MEDIUM requests to the small tier; escalate REASONING requests |
Data sensitivity | Whether the prompt or retrieved context holds regulated or confidential data | PII and classification detectors on the prompt and context | Keep requests carrying customer PII on a self-hosted model |
Caller role or identity | Who or which agent is asking, and what it may access | Identity token from your identity provider | Analysts may reach the frontier tier; a batch agent may not |
Budget | Spend remaining for the user, team, application, or agent | Spend counters checked before the call | Downshift to the small tier at 80% of the monthly budget |
Latency target | How fast the answer must return | Per-route service level and live model latency | Route interactive chat to the fastest model that clears the quality bar |
Task complexity scoring takes several forms in shipping products. LiteLLM's Auto Router v2, released in July 2026, scores each request into SIMPLE, MEDIUM, COMPLEX, and REASONING tiers with a heuristic scorer or an optional LLM classifier [11]. Amazon Bedrock Intelligent Prompt Routing predicts each model's response quality and switches from the fallback model only when another model's predicted quality is better by a margin you set [17].
Sensitivity and role are the signals most routers leave to you, and in regulated estates they often override complexity. A simple request that carries a patient record should stay on a self-hosted model even when a frontier model would answer it well. The companion guide on AI agent access control covers how identity reaches the point of enforcement.
How Does Routing Reduce Inference Cost?
Routing lowers average cost per request by moving traffic from the highest-priced tier to cheaper tiers, and the savings depend entirely on your traffic mix and how often escalation fails. A reusable way to model it:
Expected cost per request = (p₁ × c₁) + (p₂ × c₂) + (p₃ × c₃) + c_router + (e × c₃)
Variable | Meaning | How to estimate it |
|---|---|---|
p₁, p₂, p₃ | Share of requests answered at the deterministic, small-model, and frontier tiers (sum to 1) | Label a sample of real traffic by the cheapest tier that answers it correctly |
c₁ | Cost of a deterministic answer: query compute, zero tokens | Query engine cost per query |
c₂, c₃ | (Input tokens × input price) + (output tokens × output price) at that tier | Provider price pages and your average token counts [2][3][4] |
c_router | Cost of the routing decision itself (classifier inference, rule evaluation) | Benchmark the router per request |
e | Share of requests that fail at a cheaper tier and are retried at the frontier tier | Measure from escalation logs after launch |
The arithmetic shows the size of each lever. A request with 2,000 input tokens and 500 output tokens costs $0.045 at gpt-6-astra list prices and $0.00045 at gpt-6-luna list prices [3]. Every request that moves down a tier closes most of that gap, and every failed attempt that escalates pays for both tiers, which is why e matters as much as p.
Cascading is the pattern of trying a cheaper model first and escalating only when the answer fails a quality check. The research results are encouraging but measured on public benchmarks, so treat them as evidence that the approach works and measure your own savings separately. FrugalGPT reported matching the best individual model with up to 98% cost reduction [5]. Hybrid LLM, presented at ICLR 2024, reported up to 40% fewer calls to the large model with no drop in response quality [8]. RouteLLM reported cost reductions of over two times "in certain cases" without compromising quality [6]; its authors' blog put the figures at over 85% on MT Bench, 45% on MMLU, and 35% on GSM8K while keeping 95% of GPT-4's performance [7]. RouteLLM routed between only two models, and its authors list N-way routing as future work [6].
NexusOne's view is that the deterministic tier produces the largest single reduction for enterprise data questions, because a share of what reaches models today has an exact answer in a governed table. That claim has not been tested in the EMA survey, and you should measure p₁ on your own traffic before relying on it. The NexusOne guide to AI token cost control covers budgets and caps that sit alongside routing.
How Is Model Routing Different From an LLM Gateway?
An LLM gateway is a proxy that gives applications one API across many model providers and handles keys, rate limits, logging, and spend tracking, while a router decides which model a request should go to. The two often ship together. LiteLLM describes its proxy as a self-hosted, OpenAI-compatible gateway with virtual keys, per-team budgets, spend tracking, guardrails, and caching [9], and its router adds load balancing, fallbacks, latency-based and cost-based strategies [10].
Dimension | LLM gateway | Model router |
|---|---|---|
Core job | Connect, authenticate, meter, and log model calls | Choose the target for each request |
Decision basis | Configuration: which model the caller named, rate limits, fallbacks on error | Request content plus policy: complexity, sensitivity, role, budget, latency |
Typical "routing" | Load balancing across deployments of the same model; failover between providers [10][13] | Choosing between different models, or a non-model system, per request [11][20] |
Cost control | Caps and reports spend after the caller picks a model | Lowers the price of each request by picking a cheaper target |
Failure mode | Every request still reaches the model the developer hardcoded | A misjudged request gets a weaker answer or escalates and pays twice |
If your applications already call a gateway, adding a router usually means adding one routing rule set behind it, so the clients do not change.
Which Platforms Support Routing to SLMs vs Frontier Models?
Several products route between smaller and larger models on task complexity, and they differ most on whether they run inside your estate and whether they can answer without a model. Capabilities below come from each vendor's documentation as of October 20, 2026; check current documentation before you decide.
Platform | How it routes | Can it route to self-hosted SLMs? | No-model (deterministic) tier | Deployment | Better fit when |
|---|---|---|---|---|---|
LiteLLM | Strategies including latency-based and cost-based; Auto Router v2 tiers requests SIMPLE to REASONING [10][11] | Yes, any endpoint it supports among 100+ LLM providers [9]; check current documentation for your serving stack | Not described in documentation | Open source (MIT), self-hosted [9][10] | You want an open-source gateway and will own the routing rules |
OpenRouter | Auto Router classifies about 30 task types and ranks models by trailing 7-day spend share within a cost tier [12] | Hosted catalog only; check current documentation | Not described in documentation | Hosted service; you pay the selected model's rate with no router fee [12] | You use hosted models and want a broad catalog with price and latency sorting [13] |
Martian | Setting the model to "router" picks the model expected to perform best; max_cost and willingness_to_pay tune the trade-off [14] | Check current documentation | Not described in documentation | Hosted API [14] | You want a managed router tuned on a per-request cost ceiling |
Not Diamond | Recommends a model per request; custom routers train on your prompts, model responses, and evaluation scores [15][16] | Yes, it supports routing between custom models; check current documentation | Not described in documentation | Hosted recommendation API; your code makes the model call ("Not Diamond is not a proxy") [15] | You have evaluation data and want routing trained on your own quality metric |
Amazon Bedrock Intelligent Prompt Routing | Predicts response quality and routes between two models in one family using a quality-difference threshold [17] | Within Bedrock model families only: Amazon Nova, Meta Llama, and Anthropic Claude [17][19] | Not described in documentation | AWS managed, GA April 22, 2025 [18] | Your workloads already run on Bedrock and English prompts dominate (the router is optimized for English) [17] |
Azure AI Foundry model router | A trained ML model scores each prompt and picks from a pool in Balanced, Cost, or Quality mode [20] | Within the Foundry pool only, which spans OpenAI, Anthropic, xAI, DeepSeek, Meta, and others and can be narrowed to a subset [21] | Not described in documentation | Azure managed; billed as the sum of underlying model costs [21] | You are standardized on Azure and want routing with no infrastructure to run |
vLLM Semantic Router | Signal-driven routing across local, private, and frontier models, with jailbreak and sensitive-leakage detection [22] | Yes | Not described in documentation | Open source (Apache-2.0), self-hosted [22] | You run vLLM and want an open-source router to build on |
NexusOne AI Data Control Plane | Identity first, then tiered routing from query engines to small models to frontier models | Yes | Yes, deterministic questions answered by query engines at zero tokens | Customer-controlled Kubernetes on-prem, cloud, hybrid, or air-gapped | Your requests touch governed enterprise data across several systems |
Vendor savings claims vary in how they are measured. AWS states that Intelligent Prompt Routing "can reduce costs by up to 30% without compromising on accuracy" [19], and the Bedrock documentation notes that routing may not be optimal for specialized use cases [17]. Azure's documentation states that managed routing does not remove the need for evaluation and recommends re-evaluating after any change to mode or model subset [20].
When Should You Skip Model Routing?
Routing pays off when traffic is high-volume and mixed in difficulty, and it adds cost and risk when traffic is low or uniformly hard. If one application sends a few thousand requests a month, the engineering time to build an evaluation set and tune thresholds will exceed the token savings. If nearly every request needs frontier reasoning, such as complex code generation, a router mostly adds latency and an extra failure point. And if you need one model for every turn of a conversation, for reproducibility or prompt-cache hits, Azure's documentation itself advises a direct deployment over the router [20].
The strongest objection is quality drift: a router that saves money by quietly sending hard requests to a weaker model degrades answers without anyone noticing. The answer is to log which model handled each request and score a sample of responses continuously, which Amazon Bedrock and Azure both return with each response [17][20]. Run the router in shadow mode first, compare its choices against your current model, and set the escalation threshold from that data.
How NexusOne Approaches Model Routing
NexusOne calls this intelligent AI query routing, and it runs inside the NexusOne AI Data Control Plane as the first step of every AI and agent request. The control plane identifies the caller first, using one identity and policy model (Keycloak and Apache Ranger) that governs people, jobs, and agents alike. It then screens the request for jailbreak attempts and PII at the edge, before any model sees it.
Routing follows a fixed order. Deterministic questions go to the query engines (Trino, Apache Spark, and Apache Kyuubi over federated sources) and are answered at zero tokens. Bounded tasks go to small models, and only requests that need frontier reasoning reach a frontier model. Each request is attributed to the person or agent that made it, with per-role and per-agent budgets and circuit breakers that stop spend at a limit, and every allowed and blocked request is logged with the reason for the decision.
The control plane runs identically on-prem, in the cloud, hybrid, or air-gapped on a Kubernetes cluster you control, so sensitive requests can stay on your own GPUs. It ships as 85+ integrated open-source components under NexusOne's published 5-5-5-5 deployment model: about 5 minutes to provision, 5 hours to deploy, 5 days to the first governed workload, and 5 weeks to enterprise production, with Embedded Builders working alongside your team. NexusOne's view is that the zero-token tier and per-principal budgets together make AI cost predictable; measure it on your own traffic. See the AI Data Control Plane for details, or book an expert consultation.
Key Takeaways
Model routing sends each request to the cheapest target that meets quality, latency, and policy limits, including a query engine that answers deterministic questions with zero tokens.
Output prices for the highest- and lowest-priced text models from one provider differed by up to 100 times at October 2026 list prices [2][3].
Route on five signals: task complexity, data sensitivity, caller role, budget, and latency; complexity is the one most products score automatically.
Model expected cost with tier shares, per-tier token cost, router overhead, and the escalation rate, and measure each on real traffic.
A gateway connects and meters model calls; a router chooses the target. Most enterprises need both.
Managed cloud routers stay inside one cloud's model catalog; self-hosted and data-aware options fit estates with sensitive data across several systems.
Frequently Asked Questions
What Is Model Routing and How Does It Help Enterprises Pick the Right AI Model per Request?
Model routing places a decision step in front of your models that picks a target for each request based on task complexity, data sensitivity, the caller's role, budget, and latency. Deterministic questions go to a query engine, bounded tasks to a small model, and hard reasoning to a frontier model. The application calls one endpoint, so models can change without client code changes.
How Does Intelligent AI Model Routing Reduce LLM Inference Costs for Enterprise Workloads Without Sacrificing Quality?
Routing moves requests that a cheaper model answers correctly away from frontier models, which cost up to 100 times more per output token at October 2026 list prices. Quality holds when you set escalation thresholds from an evaluation set, log which model answered, and score samples continuously. Research such as RouteLLM and Hybrid LLM reported large reductions in frontier calls without measured quality loss on public benchmarks.
How Does Intelligent AI Query Routing Reduce LLM Inference Costs for Enterprises?
Query routing adds a tier that model-only routers lack: questions with an exact answer in a governed system go to a query engine and consume zero tokens. The remaining requests are split between small and frontier models. The savings depend on how much of your traffic is deterministic, so label a sample of real requests by the cheapest tier that answers each one correctly before you estimate.
Which Platforms Support Routing AI Queries to SLMs vs Frontier Models Based on Task Complexity?
LiteLLM (Auto Router v2), OpenRouter (Auto Router), Martian, Not Diamond, Amazon Bedrock Intelligent Prompt Routing, Azure AI Foundry model router, and the open-source vLLM Semantic Router all route on request complexity or predicted quality. LiteLLM, vLLM Semantic Router, and NexusOne can route to self-hosted small models, while the cloud routers stay within their own model catalogs. NexusOne also answers deterministic questions from query engines.
What Is the Difference Between Model Routing and an LLM Gateway?
An LLM gateway is a proxy that gives applications one API across model providers and manages keys, rate limits, logging, and spend tracking. A model router decides which model, or which non-model system, should answer each request. Gateways cap spend after a developer picks a model; routers lower the price of each request by picking a cheaper target. Many products, LiteLLM among them, combine both.
References
More Tools, Less Control: The 2026 State of the Data Estate Report, NexusOne, published September 30, 2026 (survey conducted by Enterprise Management Associates (EMA), September 2026, n=202 data and analytics decision-makers at organizations with 500+ employees in the US, Canada, the UK, and France). https://nexus0ne.substack.com/p/more-tools-less-control-the-2026 (retrieved October 20, 2026)
Claude Pricing (API), Anthropic. https://claude.com/pricing (retrieved October 20, 2026)
API Pricing, OpenAI. https://developers.openai.com/api/docs/pricing (retrieved October 20, 2026)
Gemini Developer API Pricing, Google AI for Developers. https://ai.google.dev/gemini-api/docs/pricing (retrieved October 20, 2026)
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, Lingjiao Chen, Matei Zaharia, James Zou, arXiv, May 9, 2023. https://arxiv.org/abs/2305.05176 (retrieved October 20, 2026)
RouteLLM: Learning to Route LLMs with Preference Data, Isaac Ong et al., arXiv, June 26, 2024 (v4 February 23, 2025; ICLR 2025). https://arxiv.org/abs/2406.18665 (retrieved October 20, 2026)
RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing, LMSYS Org, July 1, 2024. https://lmsys.org/blog/2024-07-01-routellm/ (retrieved October 20, 2026)
Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing, Dujian Ding et al., ICLR 2024, arXiv, April 22, 2024. https://arxiv.org/abs/2404.14618 (retrieved October 20, 2026)
LiteLLM Documentation (Getting Started and LLM Gateway), BerriAI. https://docs.litellm.ai/docs/ (retrieved October 20, 2026)
Router: Load Balancing, Fallbacks, LiteLLM Documentation, BerriAI. https://docs.litellm.ai/docs/routing (retrieved October 20, 2026)
Auto Router v2, LiteLLM Blog, Krrish Dholakia, July 13, 2026. https://docs.litellm.ai/blog/autorouter-v2 (retrieved October 20, 2026)
Auto Router, OpenRouter Documentation. https://openrouter.ai/docs/guides/routing/routers/auto-router (retrieved October 20, 2026)
Provider Routing, OpenRouter Documentation. https://openrouter.ai/docs/features/provider-routing (retrieved October 20, 2026)
Hello World, Martian Model Router Documentation. https://docs.withmartian.com/martian-model-router/getting-started/hello-world (retrieved October 20, 2026)
model_select vs. create, Not Diamond Documentation. https://docs.notdiamond.ai/docs/model_select-vs-create (retrieved October 20, 2026)
Training a Custom Router, Not Diamond Documentation. https://docs.notdiamond.ai/docs/offline-personalization (retrieved October 20, 2026)
Understanding Intelligent Prompt Routing in Amazon Bedrock, Amazon Bedrock User Guide, AWS. https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-routing.html (retrieved October 20, 2026)
Amazon Bedrock Intelligent Prompt Routing Is Now Generally Available, AWS What's New, April 22, 2025. https://aws.amazon.com/about-aws/whats-new/2025/04/amazon-bedrock-intelligent-prompt-routing-generally-available (retrieved October 20, 2026)
Amazon Bedrock Intelligent Prompt Routing, AWS. https://aws.amazon.com/bedrock/intelligent-prompt-routing (retrieved October 20, 2026)
How Model Router Works, Microsoft Foundry Documentation, Microsoft Learn, updated September 10, 2026. https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-router-how-it-works (retrieved October 20, 2026)
How to Use Model Router, Microsoft Foundry Documentation, Microsoft Learn. https://learn.microsoft.com/azure/foundry/openai/how-to/model-router (retrieved October 20, 2026)
vLLM Semantic Router (github.com/vllm-project/semantic-router), Go Packages. https://pkg.go.dev/github.com/vllm-project/semantic-router (retrieved October 20, 2026)
