How to Reduce LLM Costs by 50-60% Using Model Routing

...explained with code.

Brian Armstrong from Coinbase said humans shouldn’t be the ones choosing which model answers a request. Model selection is a job for AI.

Coinbase is already running on that belief. They built a routing layer in front of every request.

LLM routing sits between the app and model providers, reads each prompt, classifies the task, and forwards it to the right model automatically.

Coinbase used it to match each prompt to the right model and default to cheaper options where the task allowed it. Cache hit rate went from 5% to 60% and spend ended up nearly cut in half while usage kept growing.

Plano (here’s the open-source GitHub repo) is an open-source implementation of exactly that routing layer, the one that decides which model answers which request, without your application code ever specifying one.

The way it works is that you define routing preferences in plain English, and Plano orchestrator infers the domain and action of each incoming prompt, matches it to the right preference, and picks the model automatically.

Next, let’s set up Plano in front of Hermes Agent, and watch it route requests to different models based on what each prompt actually needs.

Before the setup, let’s understand one key concept.

---

The problem with routing

When an agent runs a task, it doesn’t send a single request. One user prompt triggers several LLM calls in sequence through tool use, each building on what came before.

With a hardcoded model, that’s never a problem. But once routing is in place, each of those calls gets classified independently.

Consider an agent debugging a function. It makes a planning call, runs a tool, reads the output, then makes another call to analyze what came back.

Call 1 routes to claude-sonnet-4.5 for code generation. Call 4 gets classified as analysis and routes to gpt-4o-mini instead.

Two problems occur when that happens.

• The model on call 4 has no memory of decisions made in call 1.

• And every model switch invalidates the cache, so you’re paying full token rate again from scratch.

Plano solves this with model affinity.

The first call in a session routes normally, and Plano pins that model to a session ID passed in a header.

Every subsequent call with the same ID goes to the same model regardless of how it gets classified.

import uuid
from openai import OpenAI

client = OpenAI(base_url="http://localhost:12000/v1", api_key="EMPTY")
affinity_id = str(uuid.uuid4())

response = client.chat.completions.create(
    messages=messages,
    tools=tools,
    extra_headers={"X-Model-Affinity": affinity_id}
)

The pin lasts 10 minutes by default.

When the task changes, you generate a new ID, and routing starts fresh. For multi-replica deployments, backing it with Redis keeps the pin shared across instances:

routing:
  session_ttl_seconds: 600
  session_cache:
    type: redis
    url: redis://localhost:6379

Model affinity handles consistency within a task. Now let’s build the routing layer using Plano.

Setup

Plano runs as a local proxy on your local machine. The routing config, model providers, and listener settings…everything is defined in one YAML file.

Plano supports three routing methods:

• Model-based: the client specifies an exact model name on every call.

• Alias-based: you define a friendly name like fast-model or reasoning-model that maps to a real model behind it, so your app never hardcodes a provider, and swapping models is one line in the config.

• Preference-aligned: the client specifies no model at all. Plano-Orchestrator infers the domain and action of each prompt and picks the right model automatically based on plain-English descriptions you define.

For alias-based, the config looks like this:

model_aliases:
  fast-model:
    target: gpt-4o-mini
  reasoning-model:
    target: claude-sonnet-4.5

We’ll use preference-aligned routing in this walkthrough.

First, we install Plano. Also, we’re using OpenRouter as our model provider, so one key gives us access to every model we need without managing separate API keys:

pip install planoai
export OPENROUTER_API_KEY=sk-or-...

Note: OpenRouter is one option. Plano works with any OpenAI-compatible endpoint, so you can point it at a self-hosted model running on Ollama or vLLM just by swapping the base_url.

Then we create the config.

The routing preferences tell Plano which tasks go where. For complex reasoning and code, we use claude-sonnet-4.5, for everything routine gpt-4o-mini:

# plano_config.yaml
version: v0.4.0

listeners:
  - name: egress_traffic
    type: model
    address: 0.0.0.0
    port: 12000
    timeout: 30s

model_providers:
  - model: anthropic/claude-sonnet-4.5
    base_url: https://openrouter.ai/api/v1
    access_key: $OPENROUTER_API_KEY

  - model: openai/gpt-4o-mini
    base_url: https://openrouter.ai/api/v1
    access_key: $OPENROUTER_API_KEY
    default: true

routing_preferences:
  - name: code generation
    description: writing new code, implementing functions, or building features from scratch
    models:
      - anthropic/claude-sonnet-4.5

  - name: complex reasoning
    description: deep analysis, architectural decisions, debugging hard problems, multi-step reasoning
    models:
      - anthropic/claude-sonnet-4.5

  - name: general tasks
    description: answering questions, explaining concepts, quick lookups, casual conversation
    models:
      - openai/gpt-4o-mini
tracing:
  random_sampling: 100
  opentracing_grpc_endpoint: http://localhost:4317

Plano reads every incoming prompt and routes it to the right model based on these preferences.

Pointing Hermes at Plano

Hermes connects to any OpenAI-compatible endpoint, so we point it at Plano instead of a provider directly:

First, we install Hermes:

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

Then we configure it to use Plano as its endpoint:

hermes model
# Select "Custom endpoint (self-hosted / VLLM / etc.)"
# Enter URL: http://localhost:12000/v1
# Skip API key

From here, every prompt Hermes sends goes to Plano, which classifies it and forwards it to the right model through OpenRouter. Hermes just got a completion back.

We start Plano first, so the proxy is ready before Hermes begins sending requests:

planoai up plano_config.yaml
hermes

Hermes is now sending every request through Plano instead of directly to a provider.

Routing in practice

Now that both Plano and Hermes are up, we send two different kinds of requests.

First, a simple question:

> what does the __init__ method do in Python?

Plano classifies this as “general tasks” and routes it to gpt-4o-mini. Fast, cheap, more than capable enough for a conceptual question.

Then something that actually needs horsepower:

> implement a rate limiter middleware for a FastAPI app using Redis, with per-user and per-endpoint limits

Plano classifies this as “code generation” and routes it to claude-sonnet-4.5.

The Hermes session is hitting the same endpoint, but it’s picking a different model automatically based on what the prompt actually needed.

We wrote zero branching logic to make that happen. The routing decision lived in the YAML, not in the agent.

That covers routing by intent. The next case is when two models can both handle a task equally well, and the only difference is price.

Routing to the cheapest model that qualifies

When two models can both handle a task equally well, we can let Plano pick based on live pricing instead of the order we wrote them in.

routing_preferences:
  - name: code review
    description: reviewing, analyzing, and suggesting improvements to existing code
    models:
      - anthropic/claude-sonnet-4.5
      - openai/gpt-4o
    selection_policy:
      prefer: cheapest

Plano reads a pricing catalog before routing and reorders the candidate list by total token cost.

The cheaper model answers, but the expensive one stays as a fallback if the cheaper one fails.

Plano also ships an observability console, planoai obs, which shows a per-request USD cost column so we can see exactly which model answered each request and what it cost.

Model agility, without the rewrite

Armstrong's team built their routing layer from scratch, inside custom harnesses with engineers assigned to own it.

Most teams are making the same decisions without any of that infrastructure

Plano gives you three things out of the box without writing a single line of routing logic.

• Preference-aligned routing automatically sent complex tasks to claude-sonnet-4.5 and routine ones to gpt-4o-mini.

• Cost-aware selection reached for the cheaper option when two models could both handle the job.

• Model affinity kept the same model pinned across every turn of a multi-step task, so nothing broke mid-session.

All of it is configured in YAML instead of inside your coding agent.

When a cheaper model drops for routine tasks, you update one line. When a better one beats Claude on code, you point the preference at it. The agent never changes.

The routing decision just moves.

Plano’s GitHub repo →

(Don’t forget to star it ⭐️)

Good day!