Building a Multi-Model Agent Router for Platform Engineering

The Problem: One Model Is a Bottleneck

Most AI-augmented workflows start with a single model. You pick GPT-4 or Claude or DeepSeek, wire it into your pipeline, and ship it. For a while, it works.

Then the cracks appear. The model that's great at reasoning through a complex infrastructure migration is expensive and slow for simple ticket triage. The model that's fast and cheap for routine PR reviews can't handle the nuance of a security audit. The model with vision capabilities sits idle because your pipeline never sends it screenshots. You're paying for capability you're not using — or worse, leaving capability on the table because your pipeline is hard-wired to one endpoint.

The Architecture

A multi-model agent router sits between your platform engineering agents and the LLM providers. It inspects each incoming task, classifies it, and routes it to the optimal model based on task type, priority, cost budget, and modality requirements.

The Decision Pipeline

1. Task arrives — a ticket, a PR review request, a deployment verification, a Slack command, a cron-triggered diagnostic
2. Classifier inspects the task — what kind of work is this? What's the complexity? Does it need vision? Is it latency-sensitive? Is there a cost cap?
3. Router selects the model — based on a configurable matrix of task type → model mappings
4. Model executes — the task goes to the right provider with the right context
5. Result returns — response is post-processed and delivered back to the workflow

What Gets Routed Where

This is the routing matrix we use in production at Ricotek. It's not dogma — it's a starting point tuned to our workloads. Yours will differ.

Task Type	Primary Model	Why	Fallback
Infrastructure planning, architecture decisions, security audits	Claude (Anthropic)	Best at structured reasoning, long-context analysis, and cautious recommendations	GPT-4
Code generation, PR reviews, pipeline scripting	DeepSeek	High throughput, low cost, strong code generation at scale	Claude
Screenshot analysis, dashboard triage, log visualization	Gemini (vision)	Native vision capabilities, fast image analysis	GPT-4V
Ticket triage, routine classification, low-complexity tasks	DeepSeek	Cheapest option, sufficient for classification and simple responses	Any available
Self-healing diagnostics, SRE agent checks	DeepSeek	Fast, cheap, runs 20+ checks without breaking the budget	Claude
Customer-facing responses, high-stakes communication	Claude	More natural tone, better at nuanced communication	GPT-4

Real Example: The Hermes Gateway

At Ricotek, we run a custom Hermes gateway that implements this routing pattern. The gateway is the single entry point for all agent traffic. Agents don't know or care which model handles their request — they fire a task and get a response. The gateway handles classification, routing, retries, and fallback.

Classification Strategy

We use a two-tier classification approach. The first tier is rule-based — fast, deterministic, zero-cost. It catches the obvious cases:

• Does the request contain an image? → route to vision model
• Is the task tagged security-review? → route to Claude
• Is it a cron-triggered diagnostic? → route to DeepSeek
• Is the cost budget explicitly capped? → prefer cheapest eligible model

The second tier is model-based. When rule-based classification doesn't match, a lightweight classifier (running on the cheapest available model) inspects the task and determines the complexity, modality needs, and priority. This adds ~200ms of latency but ensures tasks that don't fit a simple rule still get routed correctly.

Fallback and Resilience

Single-model architectures have a single point of failure. If your one provider is down, you're down. A multi-model router solves this implicitly: if the primary model returns a 429 or 5xx, the router retries on the fallback model automatically. We implement exponential backoff with a 3-attempt max across all providers.

The fallback chain is configurable per task type. For code generation, the chain is DeepSeek → Claude → error. For vision tasks, it's Gemini → GPT-4V → error. Critical tasks get more fallback attempts than routine ones.

What This Actually Delivers

Moving from a single-model pipeline to a multi-model router delivered three concrete outcomes in our production environment:

• 40-60% cost reduction — routine tasks hit the cheapest model, not the most capable one. Expensive reasoning models only fire when the task actually needs them.
• Better task outcomes — security audits go to the model that's best at security audits. Code generation goes to the model that's best at code. You stop settling for "good enough from one model" and start getting "best available for this task."
• Zero-downtime resilience — no single provider outage can take down the platform. If DeepSeek is degraded, Claude picks up the slack automatically. Agents don't notice.

When This Pattern Doesn't Work

A multi-model router adds complexity. Don't reach for it if:

• You have fewer than 3 distinct task types. The classification overhead isn't worth it for simple workflows.
• Your volume is low. Under ~100 agent tasks per day, the cost savings from routing don't justify the architectural complexity.
• You need consistent tone and style across all outputs. Different models produce noticeably different voices. If brand consistency is critical, stick to one model and tune it.
• Your team isn't ready to manage multi-provider credentials and rate limits. Each provider is a new operational surface — API keys, usage monitoring, billing, rate limit tracking.

Getting Started

The fastest path to a working multi-model router:

1. Map your task types. What does your platform actually do? List every distinct workflow — PR review, ticket triage, incident response, deployment verification, etc.
2. Benchmark models against your tasks. Don't guess. Run 50-100 samples of each task type through each candidate model. Measure accuracy, latency, and cost per task.
3. Start with two models. Don't build a 5-provider routing matrix on day one. Pick two — one for complex reasoning, one for throughput — and route between them. Add more as the data justifies it.
4. Track cost per task type. You can't optimize what you don't measure. Tag every request with its task type and track cumulative cost. The router should make the cost curve visibly flatter.
5. Build fallback before you need it. The first time a provider goes down during a production deployment, you'll wish you had. Implement retry-with-fallback from day one.