Smart routing: the cheapest provider that is fast enough for your prompt
Several providers sell the same model, and the cheapest one is often the slowest on long prompts. So for every request UberLLM checks how fast each provider starts answering at that prompt size and what this request would cost on each one, keeps the providers that are fast enough (within 2x of the fastest, or under 1 second), and sends the request to the cheapest of those. There is nothing to set: it runs on every chat request unless you pass your own routing options.
Why the cheapest provider was not good enough
A model like DeepSeek V4.1 Flash is sold by several providers at different prices. Sending every request to the cheapest one works for short prompts. It breaks on long ones, because before a model writes its first word it has to read the whole prompt, and some providers read long prompts far slower than others.
We measured it on live traffic on 28 October 2026. On DeepSeek V4.1 Flash, the cheapest provider took 15 seconds to send the first word for prompts of 32k–128k tokens, and 34 seconds above 128k. Another provider took 1.5–2 seconds for the same prompts.
That matters most for coding agents. Claude Code, Cline and similar tools send the whole working context on every turn, often 60k–130k tokens, and most of it is a cache read.
Price alone also picked badly on those prompts. Ranking by the normal input price ignored the price of cache reads, so a long cached prompt could go to a reseller charging $0.07 per million cached tokens when another provider charged $0.0013. That call was slower and about 50x more expensive.
What happens on every request
- Measure speed by prompt size. UberLLM records each provider’s time to first token for every model, in four size groups: under 8k tokens, 8k–32k, 32k–128k and over 128k. The same provider answered a 31-token prompt in 1.7 seconds and a 105k-token prompt in 15.5 seconds, so one number per provider would be wrong. The data comes from real calls. Separate speed checks every 5 minutes at about 20k, 60k and 150k tokens catch a slowdown at one size even when short calls are fast.
- Price this exact request on each provider. Input, cached input and output are priced separately, on this request’s mix of tokens. For long agent prompts, the cache-read price is usually most of the bill.
- Keep the providers that are fast enough. A provider qualifies when its typical time to first token is within 2x of the fastest measured provider, or under 1 second. Speed is judged on the median. Slow outliers count at a quarter weight once a provider has at least 10 samples, so one bad call does not push a good provider out.
- Send the request to the cheapest of those. The fastest provider only wins when the cheaper ones are too slow.
- Keep the data current. 5% of requests go to a provider we have too little data on, so a provider that got faster is noticed. A provider that refuses a call is treated as unusable and tried again within 15 minutes.
With no speed data for a model yet, the request goes to the cheapest available provider. A provider priced above the model’s usual price is only picked for speed when it is still at least 50% below the model’s official list price.
Long prompts: a second provider when the first is slow
Even a fast provider sometimes starts slowly. On one cached 120k-token prompt, the same provider started in 0.45, 3.7 and 8.5 seconds on three calls in a row. Choosing the right provider cannot fix that, so long streamed requests get a backup:
- Prompts of 32k–128k tokens: if no first word has arrived after 1 second, the same request also goes to a different fast provider.
- Prompts over 128k tokens: both requests start at once.
- The first provider to answer streams the reply. The other request is cancelled as soon as the first word arrives.
You get one reply and pay for one reply. We pay for the cancelled request, out of a fixed daily budget. When that budget runs out, long requests go to one provider until the next day (UTC).
Retries go to a different provider
If a call fails before any of the reply reaches you, UberLLM retries it on another route, up to 4 attempts. Before smart routing, a retry could land on the same slow provider again. A call that fails on every attempt costs $0. The details are in models and routing.
Results on live traffic
DeepSeek V4.1 Flash, time to first word:
| Prompt size | Before | After |
|---|---|---|
| 32k–128k tokens | 18 s | 3.3 s |
| Over 128k tokens | 40 s | 3.4 s |
On streamed calls over 32k tokens (28 October 2026), the slowest 10% of calls went from 57 seconds to 8.7 seconds. The cost of 1,000 of those calls went from $2.49 to $1.04, because cache-read prices now count in the choice. On 29 October a 203k-token prompt started answering in about 4 seconds.
One limit: no provider starts a 100k-token prompt instantly. It takes a few seconds on all of them. Smart routing removes the extra wait from the slow providers. It cannot remove the time a fast provider needs to read the prompt.
What you pay
Each request is billed what the provider that served it charged, with no UberLLM fee added. Every call and its cost is listed in Dashboard → Activity. To cap the price, send provider.max_price with both prompt and completion. Smart routing then only picks providers under your cap, and a request is never billed above it.
Using your own routing instead
You do not need to change anything to get smart routing. If you name providers in provider.only, provider.ignore or provider.order, set provider.sort, or add a :floor or :nitro suffix to the model id, your choice is used and smart routing does not run for that request:
{
"model": "deepseek/deepseek-v4.1-flash",
"provider": { "sort": "price" },
"messages": [{ "role": "user", "content": "Hello" }]
}
provider.max_price is the exception: it works together with smart routing. All the options are listed in models and routing.