Back to Blog
AI AgentsLLM RoutingToken Optimizationn8nArchitecture

AI Load Balancer: Route Queries to the Right Model and Cut Token Costs by 70%

Fasil Hawultie··10 min read

The Problem Nobody Talks About

You built an AI agent. It works. Users love it. Then the bill arrives.

Most teams pick a powerful model, wire everything through it, and never revisit the decision. Every query, whether it's "what's the capital of France?" or "analyse this 40-page contract for liability clauses," hits the same model at the same price.

That is the equivalent of hiring a senior consultant at $500/hour to answer your emails.

The fix is a model router: a layer that classifies each query and sends it to the cheapest model capable of answering it correctly. Simple queries go to fast, cheap models. Complex ones go to powerful ones. You pay for firepower only when you need it.

In production, this typically cuts AI costs by 60-75% and reduces average response time by 50-70%.


First: What Are Tokens?

When you send a message to an AI, it doesn't read words. It reads tokens. One token is roughly 3/4 of a word. A 1,000-word document is roughly 1,300 tokens.

You pay for every token in (your question + context) and every token out (the AI's answer). More tokens = higher cost + longer wait.

Here is what different models charge for the same token, as of mid-2026:

ModelInput per 1M tokensOutput per 1M tokens
GPT-4o mini~$0.15~$0.60
Claude Haiku 4.5~$0.80~$4.00
GPT-4o~$2.50~$10.00
Claude Sonnet 5~$3.00~$15.00
Claude Opus 4.8~$15.00~$75.00

Always check the provider's pricing page. These numbers change.

The intelligence gap between tiers is real but often irrelevant. Haiku 4.5 answers a simple lookup just as well as Opus 4.8. The difference only shows on hard tasks. That is what the router exploits.


The Three Tiers

Think of it like hospital triage. A triage nurse looks at each patient and decides: nurse practitioner, doctor, or specialist. Most cases go to the nurse practitioner. Specialists only see the rare, complex cases.

Tier 1: Fast (the nurse practitioner)

  • Models: Claude Haiku 4.5, GPT-4o mini, Gemini 2.0 Flash
  • Response time: 100-300ms
  • Best for: single-fact lookups, yes/no decisions, short summaries, formatting, classification, date extraction, translation
  • Target: ~65-75% of total queries

Tier 2: Standard (the doctor)
  • Models: Claude Sonnet 5, GPT-4o, Gemini 1.5 Pro
  • Response time: 400-900ms
  • Best for: multi-step reasoning, structured output with logic, documents under 15,000 tokens, comparing options, writing with nuance
  • Target: ~20-30% of queries

Tier 3: Power (the specialist)
  • Models: Claude Opus 4.8, OpenAI o1, top-tier from any provider
  • Response time: 1-5 seconds
  • Best for: complex multi-document analysis, deep reasoning chains, legal/financial/technical documents, anything that failed on Tier 2
  • Target: under 5-10% of queries

The goal is not to send everything to Tier 1. Send each query to the minimum tier that answers correctly. Routing a complex query to Haiku to save money, getting a wrong answer, and then having to re-run it costs more than just using Sonnet the first time.

How the Router Decides

Three signals determine the tier:

1. Query complexity

One thing or many things? Direct lookup or reasoning across multiple inputs?

  • "What year was the Eiffel Tower built?" → Tier 1. One fact.
  • "Compare these three pricing plans and recommend the best one for a startup under 10 employees" → Tier 2. Multiple inputs, a judgment call.
  • "Read this 30-page contract, identify all liability clauses, and rank them by severity" → Tier 3. Long document, domain expertise, nuanced output.

2. Context size

Even a simple query becomes expensive attached to a large document. Length overrides complexity.

Context sizeMinimum tier
Under 3,000 tokens (~2,000 words)Tier 1 eligible
3,000-15,000 tokensTier 2 minimum
Over 15,000 tokensTier 3 minimum

3. Output requirements

A one-word answer? Tier 1. A structured JSON object with 15 fields that a downstream system will parse? Tier 2 minimum. You need the model to follow the schema reliably.


Building It in n8n (No Code Required)

Here is a complete n8n workflow. No code needed for the routing logic.

The workflow shape:

[Trigger] → [Token Counter] → [IF node] → [Classifier] → [Switch] → [3 AI nodes] → [Merge] → [Respond]

Step 1: Trigger

Use a Webhook node (requests from your app) or a Chat Trigger node (chatbot). This receives the user's query.

Step 2: Token counter (the context-size override)

Add a Code node before the classifier. This is the hard override. Large inputs skip the classifier entirely.

const query = $input.first().json.query || '';
const context = $input.first().json.context || '';
const combined = query + ' ' + context;
// 1 token is roughly 4 characters
const estimatedTokens = Math.ceil(combined.length / 4);

let forcedTier = null;
if (estimatedTokens > 15000) forcedTier = 'power';
else if (estimatedTokens > 3000) forcedTier = 'standard';


return [{ json: { ...($input.first().json), estimatedTokens, forcedTier } }];

Step 3: IF node (skip classifier for large inputs)

Add an IF node:

  • If forcedTier is not empty → jump straight to the Switch node
  • If forcedTier is empty → proceed to the Classifier

Step 4: Classifier AI node

Add a Basic LLM node. Configure it:


You are a query complexity classifier. Return ONLY a JSON object. Nothing else.


Format: {"tier": "fast", "reason": "one sentence"}


Rules:
  • "fast": single fact, yes/no, simple formatting, short summary under 100 words, translation, date extraction
  • "standard": multi-step reasoning, comparing options, writing with nuance, documents under 15000 tokens
  • "power": documents over 15000 tokens, legal/financial analysis, expert-level judgment required

Return JSON only.

  • User message: {{ $json.query }}

Output looks like: {"tier": "fast", "reason": "Single-fact lookup question."}

Step 5: Switch node

Add a Switch node. Three branches:

  • Branch 1: {{ $json.tier }} equals fast
  • Branch 2: {{ $json.tier }} equals standard
  • Branch 3: {{ $json.tier }} equals power

Step 6: Three AI Agent nodes

One AI Agent node per branch. Same system prompt, same user query. Only the model changes:


Step 7: Merge and respond

Add a Merge node set to "Combine: Multiplex". Connect all three branches. Feed the output to your Respond to Webhook node.

That is the complete no-code router.


The Same Pattern in Make.com

If you use Make.com instead of n8n, the structure is identical:

  1. Webhook module → receives the query
  2. HTTP module → calls the Anthropic or OpenAI API with Haiku/mini and your classifier prompt
  3. JSON Parse module → extracts the tier field
  4. Router module → Make's Switch equivalent, branches on tier value
  5. Three HTTP modules → one per branch, each calling the appropriate model API
  6. Output module → returns the result

The blunt truth: Make.com's Router module and n8n's Switch node do the same thing. Pick whichever platform you already use.

Real Numbers: What This Saves

A support agent handling 5,000 queries per day, before routing:

  • All queries → GPT-4o
  • Avg query: 500 tokens in, 200 tokens out
  • Daily cost: 5,000 x 700 tokens x ($5/1M) = ~$17.50/day

After routing (typical split: 70% fast, 25% standard, 5% power):
TierQueriesCost
GPT-4o mini (fast)3,500 x 700 x $0.30/1M$0.74
GPT-4o (standard)1,250 x 700 x $5.00/1M$4.38
Claude Opus 4.8 (power)250 x 700 x $22.50/1M$3.94
Total$9.06/day

That is a 48% reduction. In real systems where the fast tier handles more volume, 65-75% is typical.

The latency improvement matters as much as the cost. Tier 1 responses arrive in 100-300ms. Users experience the agent as much faster even though only 70% of queries changed.


What Can Go Wrong

The classifier picks the wrong tier. When in doubt, round up. Routing a "fast" query to Standard wastes a little money but is never wrong. Log the tier per query so you can see where misclassification concentrates.

Users notice inconsistency. If the same user gets different depth of answer across sessions (Haiku one time, Opus the next), it feels broken. Fix: once a session has escalated to a tier, do not downgrade below it for the rest of that session.

The classifier adds latency. Classification takes 100-200ms. If your Tier 1 responses take 150ms, you have doubled the latency for fast queries. Fix: cache the tier classification by query. Same or similar query comes in again, reuse the decision. A Supabase table or Redis lookup in n8n handles this with no extra infrastructure.

Accuracy drops on the edge. Queries on the boundary between tiers are the hardest to classify. Build a fallback: if a Tier 1 response contains "I'm not sure" or "I don't have enough information," automatically re-run on Tier 2. Latency penalty only hits the cases that actually need it.


When NOT to Use Routing

Skip it if query volume is low. Under a few hundred queries per day, the setup and maintenance cost outweighs the savings. Just pick a mid-tier model and move on.

Skip it if complexity is uniform. If 95% of queries are genuinely hard (legal document analysis, financial modelling), the classifier sends almost everything to Tier 3 anyway. Not worth the overhead.

Skip it if consistency beats cost. Some applications need every response to feel equally authoritative. A legal document review tool probably should not silently switch between models.

Use it when: you have mixed query complexity at volume. That is where the savings are real and users notice the speed.


TL;DR: The Cheat Sheet


  1. The core idea: classify each query with a cheap model, route to the minimum model that answers correctly
  2. Tier 1 (65-75% of queries): GPT-4o mini or Claude Haiku 4.5. Fast, cheap, good enough for simple tasks.
  3. Tier 2 (20-30%): GPT-4o or Claude Sonnet 5. Multi-step reasoning, structured output.
  4. Tier 3 (under 5%): Claude Opus 4.8 or OpenAI o1. Long documents, complex analysis only.
  5. Context size overrides complexity: over 15,000 tokens always goes to Tier 3, no classifier needed
  6. n8n setup: Webhook → Code (token count) → IF → Basic LLM classifier → Switch node → 3 AI Agent nodes → Merge
  7. Typical savings: 48-75% cost reduction, 50-70% latency improvement

Links and Resources



Production-tested. Not theoretical. This is what actually works.

Want to work together?

I help businesses build AI automation solutions that deliver real results.

Get in Touch
Jazz while you're here?