Meet the Model Router -- Gemini 3.5 Pro Landed Today and Pushed the Live Model Count
racing to control the layer beneath the model -- the chips, the governance rules, the routing decisions -- because that's where the actual cost and control now live, not on the leaderboard.
RouteLLM: the open-source router that turned model-picking into a
RouteLLM's GitHub repo is getting a fresh wave of attention this week, as teams staring down seven live price tiers go looking for something better than hand-picking a model per project. Its pitch is almost aggressively simple: train a lightweight classifier on human preference data, then let it decide, for each incoming prompt, whether a cheap model will do or a strong one is worth the money. The ICLR 2025 result behind it holds up -- 85% lower cost on MT Bench while keeping 95% of GPT-4 Turbo-level quality, sending only 14% of traffic to the expensive model. It's popular for the same reason OpenRouter's Auto Router is spreading through developer circles this week: neither one asks a team to become a routing-research lab first. One is open-source and self-hosted, the other is a hosted dial -- but both let you stop reading
1) Seven tiers is one too many to pick by hand
Gemini 3.5 Pro arrives after Google reportedly scrapped an earlier version outright, after engineers found structural failures in recursive tool-calling and SVG generation, and rebuilt it on an entirely new pretraining run. The result ships a 2-million-token context window -- double Gemini 3.5 Flash's 1 million -- a Deep Think reasoning layer for multi-step problems, and autonomous workflow capability, at a reported $12-15 per million input tokens and $36-45 per million output tokens: roughly double Sonnet 5's introductory rate and in the same range as
That lands on top of a field that was already crowded. GLM-5.2, DeepSeek V4, and Qwen 3.6 shipped inside the same ten-day window Day 110 covered; Sonnet 5, Grok 4.5, and GPT-5.6 shipped the ten days before that. Seven models, seven different specialties, none of them dominant at everything -- which means benchmarking every new release against your own workload before every project no longer scales. That's exactly the gap a routing layer exists to
2) How a routing layer actually decides
RouteLLM, built by the LMSYS team behind Chatbot Arena, trains lightweight classifiers -- a matrix-factorization model and a BERT-based variant -- on human preference data to predict which model wins a given prompt, then routes accordingly. It's open-source, peer-reviewed at ICLR 2025, and self-hosted, which means the routing logic lives in your own infrastructure rather
OpenRouter's Auto Router, powered by NotDiamond, takes the hosted route: it analyzes each incoming prompt, selects a model from a curated pool, forwards the request, and returns metadata showing which model actually answered. It pins the chosen model and provider for the rest of a conversation to keep behavior consistent and prompt-cache hits intact, charges no fee of its own beyond the selected model's standard rate, and exposes a single cost_quality_tradeoff dial from 0 (always the most capable model) to 10 (always the cheapest). Martian takes a third approach -- a proxy layer that routes in real time behind one model-agnostic endpoint, so the application never has to know which model actually ran. The frontier is already moving past static classifiers: bandit-feedback online-learning routers like BaRP and PILOT adapt from live traffic instead of a training set frozen at launch, MCP Gateways are emerging as a single control plane in front of an entire tool-plus-model stack, and vLLM's Iris project is pushing the routing decision below the application layer entirely, into the inference
3) The catch: routing is a new dependency, not a free lunch
RouteLLM's 85% figure is real but pair-specific -- GPT-4 Turbo versus Mixtral 8x7B, scored on MT Bench. Run the same idea with a simpler BERT classifier on MMLU instead, and the savings drop to 45%. Martian's advertised "up to 98%" is explicitly an upper bound, not a typical result. None of these numbers travel to your workload unmodified; they're proof the technique works,
A router that misjudges a prompt doesn't throw an error -- it silently sends a hard question to a cheap model and returns a worse answer with total confidence. That makes the router's own accuracy something you now have to monitor, on top of monitoring the models it's choosing between. It also adds a dependency to the request path: if a hosted router's curated model pool or classifier goes stale, quality can drift without a single line of your own code changing. In production, teams that actually tune a routing layer report bill cuts in the 40-85% range -- a wide band, because it depends entirely on how much of your traffic ever needed the expensive model
RouteLLM (LMSYS) Open-source router 85% cost cut (MT Bench, Self-hosted, 95% GPT-4 Turbo quality) research-grade routing OpenRouter Auto Router Hosted router $0 routing fee -- pays One-dial cost/quality (NotDiamond) selected model's rate tuning (0-10) Martian Hosted proxy router Up to 98% claimed (upper Real-time routing with
Gemini 3.5 Pro (ref., new Frontier model $12-15 in / $36-45 out per 2M context, Deep Think
Today is also the opening day of Shanghai's World AI Conference, and for the first time since it launched in 2018, Xi Jinping is delivering the keynote in person -- China's clearest signal yet that it wants to help write AI's governance rules, not just follow them, including a push for a Shanghai-headquartered World AI Cooperation Organization as a counterweight to Western-led governance bodies. It lands the same week Anthropic confirmed early talks with Samsung over custom Claude inference chips, aimed squarely at its own $1.25-billion-a-month compute bill. Put next to today's topic, it's the same pattern routing exists to manage: every lab, East or West, is racing to control the layer beneath the model -- the chips, the governance rules, the routing decisions -- because that's where the actual cost and control now live, not on the leaderboard.
RouteLLM's 85% figure is real but specific to one model pair on one benchmark; a BERT-based version of the same idea only cleared 45% on a different one. Test any router against a sample of your actual prompts before trusting its savings number.
Tools like OpenRouter's Auto Router expose a single cost_quality_tradeoff parameter (0-10) instead of hiding the decision entirely. Keep that kind of manual override while you build confidence in a router's judgment, rather than adopting one you can't audit.
Gemini 3.5 Pro is this week's new price tier; six labs shipped inside the last two weeks alone. Whatever routing setup you build should assume the model list keeps growing, not that today's seven options are the final list.