AI Routing: The Layer Between You and the Models

Disclaimer: I create this content entirely on my own time, and the views expressed here are mine alone (not my employer’s). Because I love leveraging new tech, I use AI tools like Gemini, ChatGPT, Claude, Perplexity and others as a “digital team” to help research and polish these articles so I can share the best possible insights with you!

For the last few years, much of the AI conversation has focused on models.

Which model is smartest? Which one has the largest context window? Which one is fastest? Which model is best for coding, reasoning, images, research, or writing?

Those questions still matter, but I think we may be entering a stage where another question becomes just as important: Who decides which model gets the work in the first place?

As the number of models, agents, tools, and AI services continues to grow, applications will increasingly need something sitting between the user and all of those capabilities. That layer will decide where a request goes, what resources it needs, what it is allowed to access, how much it should cost, and potentially whether the work should happen locally, in the cloud, or somewhere in between.

We often call this routing, but I think what is emerging is much bigger than a simple AI traffic cop.

From One Model to Many

The first generation of generative AI applications was relatively straightforward. A user entered a prompt, the application sent it to a model, and the model returned a response.

Conceptually, it looked something like this:

User → Model → Response

That architecture made sense when most organizations were experimenting with one or two large language models. Today, the landscape is becoming considerably more complicated.

An organization might have access to several commercial models, smaller specialized models, open-source models running in its own infrastructure, local models running on PCs or edge devices, internal retrieval systems, enterprise applications, APIs, and increasingly autonomous or semi-autonomous agents.

The architecture begins to look more like this:

User → Decision/Orchestration Layer → Model / Tool / Agent / Local AI / Cloud AI → Verification → Response

The interesting part is that the user may never know which model actually performed the work.

And increasingly, they may not need to.

The Best Model May Not Be the Right Model

We have spent a lot of time benchmarking AI models based on intelligence and capability, but production systems have more constraints than benchmark scores.

A highly capable frontier model might be unnecessary for summarizing a short document. A smaller model may complete that task faster and at a fraction of the cost. A sensitive document might need to remain on a local device regardless of which cloud model performs better on a benchmark. A coding request may go to one model while image analysis goes to another. A routine customer-service question may use an inexpensive model, while an unusual or high-risk request is escalated to a more capable one.

The routing decision can potentially consider many things:

Capability and task type
Cost and token usage
Latency
Privacy and data sensitivity
Security policies
Model availability
Context-window requirements
Local versus cloud execution
Regulatory or geographic restrictions
Tool and data access
Reliability and confidence
Human approval requirements

Suddenly, asking which model is best? becomes a much less useful question.

The better question may be: Which model is appropriate for this particular task, under these particular circumstances?

Routing Is Really a Decision Layer

This is why I think the word routing can actually undersell what is happening.

Traditional routing usually means deciding where something should go. AI orchestration can involve deciding not only where a task goes, but also how it should be performed.

A system might first classify a request, determine whether sensitive information is involved, choose a model, retrieve information from an internal knowledge source, allow an agent to use a business application, ask another model to verify the result, and then determine whether a human needs to approve the final action.

That is not simply routing.

It is a decision layer.

As agents become more capable, this layer becomes even more important because the system is no longer deciding only who answers the question. It may also be deciding who is allowed to do something.

There is a significant difference between allowing an AI system to summarize an email and allowing it to send one. There is an even larger difference between asking an agent to analyze an invoice and allowing it to initiate a payment.

The orchestration layer becomes one of the places where those boundaries can be defined and enforced.

Local AI Makes This Even More Interesting

This becomes particularly important as AI capabilities move onto PCs, phones, edge devices, and private infrastructure.

I do not think the future is simply cloud AI versus local AI. It is much more likely to be a combination of both.

Some tasks will make sense in the cloud because they require large models or substantial computing resources. Others will make more sense locally because they are simple, repetitive, latency-sensitive, private, or need to work without a network connection.

Imagine an AI assistant receiving a request and automatically determining that the first part can run on the NPU in your laptop, another portion requires access to a corporate knowledge system, and a particularly complex reasoning task should be sent to a cloud model.

From the user’s perspective, it may simply appear that the AI completed the task.

Behind the scenes, several different systems may have participated.

That is where intelligent routing and orchestration become extremely valuable.

Models Will Continue to Change

There is another reason I think this layer matters.

Models are changing incredibly quickly.

Today’s leading model may not be the leading model six months from now. Pricing changes. Context windows grow. Specialized models appear. Open-source models improve. Hardware becomes more capable. New providers enter the market, while existing providers introduce entirely new architectures.

Building an application tightly around one model can therefore create another form of technology dependency.

An orchestration layer creates the possibility of abstraction.

Instead of building the application around a specific model, the application can be built around capabilities and policies. The system can then choose among available models based on what is needed at that moment.

That does not eliminate vendor lock-in. Orchestration platforms themselves can become another form of dependency, and models do not always behave identically even when APIs look similar.

But it can change where the dependency lives and potentially give organizations considerably more flexibility.

Cost Could Become a Major Driver

There is also a simple economic reason this matters: not every AI request deserves the most expensive model.

As organizations move from AI experimentation into large-scale deployment, millions of small decisions about tokens, inference, compute, and latency begin to matter.

A sophisticated system could use inexpensive models for routine work and escalate only when necessary. It could use local inference when appropriate, cache common results, select models based on current pricing, or route workloads based on performance requirements.

We already do versions of this throughout computing.

We use caches before databases. We use different storage tiers. We balance traffic across servers. We move workloads between local infrastructure and cloud infrastructure.

AI is beginning to develop its own version of that infrastructure.

The Invisible Layer May Become the Important One

This is why I think some of the most interesting AI companies may not necessarily be the ones building the largest models.

There is an entire ecosystem developing around gateways, orchestration, observability, security, agent management, model selection, evaluation, and policy enforcement.

These systems sit in an interesting position. They can potentially see the request, understand the available resources, apply organizational policies, select the appropriate intelligence, monitor what happened, and determine what happens next.

That is a strategically important place to be.

It is also why I would not describe routing simply as plumbing. Plumbing sounds mundane until you realize that everything depends on it working.

Stop Asking Only Which AI Is Smartest

I recently wrote about why I think we need to stop asking only how smart AI is. This is another reason why.

Individual model intelligence will continue to matter, and the competition between model providers will continue. But increasingly, the intelligence of the overall system may matter more than the intelligence of any individual model inside it.

The important questions become broader.

Which model should handle this task? Should the work happen locally or in the cloud? What information should the model be allowed to see? Which tools can an agent use? How much should the task cost? When should another model verify the answer? When should a human remain in the loop?

Those are architecture questions, governance questions, security questions, and increasingly business questions.

The lasting advantage may not come from choosing the best model.

It may come from building an architecture that can keep choosing the right model, tool, agent, data source, or execution environment as the landscape changes.

And perhaps that is where one of the most important layers of the emerging AI ecosystem is quietly being built: not inside the models themselves, but between them.

Articles and Further Reading

Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey — Yasmin Moslem and John D. Kelleher provide a useful survey of approaches for dynamically selecting among multiple LLMs based on query difficulty, uncertainty, specialization, cost, and efficiency.
A Unified Approach to Routing and Cascading for LLMs — Research presented at ICML 2025 examining routing, where a model is selected for a request, and cascading, where requests can move to increasingly capable models when necessary.
Cloudflare AI Gateway: Dynamic Routing — A practical example of the routing layer discussed in this article, including conditional model selection, fallbacks, rate limits, budget limits, and routing based on request metadata.
Cost-Aware Model Routing Through Vercel AI Gateway — A hands-on example of routing requests between lower-cost and more capable models based on task difficulty, verification, budgets, and escalation rules.
LangGraph Documentation — LangGraph illustrates the broader orchestration side of the discussion, providing infrastructure for stateful, multi-step agent workflows rather than simply selecting a model.
LangChain Agent and Multi-Agent Tutorials — Examples of routing work among specialized agents, tools, knowledge sources, and workflows, showing how routing expands beyond model selection into orchestration.

Videos

AI Model Routing: How Smart Routing Picks the Right LLM — Databricks — Explains how intelligent routing can select different models based on the request while balancing capability, latency, and cost.
LLM Routing Explained with 3 Examples, Simple to Advanced — echohive — A more hands-on explanation showing several approaches to routing requests between models and workflows.