> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models.
I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially good at is more or less just throwing spaghetti against the wall, except the spaghetti is potentially very expensive and might insert subtle issues into your code base.
drusepth 15 hours ago [-]
One of the best features I built into our game studio's AI router (IMO) was to take a random sampling of generations and have them generate across ALL models (that we support, at least) so we, the developers, can browse outputs and get a sense each model's output. Seeing them all side by side for generations you're already familiar with makes it feel like significantly less cognitive load.
The better I get at recognizing what each model is good/bad at, the more I'm glad we're taking the time to choose specific models for specific prompts -- and the more I wouldn't trust a generic router to efficiently route for me.
fennecbutt 10 hours ago [-]
This is a problem we're looking at atm. We want to start running evaluations as we're having problems with consistency across releases even just for individual agents.
cyanregiment 20 hours ago [-]
There are "Hi" and "Lo" variants of the different models.
Maybe Gemini or ChatGPT is a "Hi" while running a local Mistral 7b is a "Lo" - sometimes you want lo-res technologies. Faster, lighter, does the job.
Find the same with image, video, vision, and so-on.
Probably get away with pretty weak vision models for some basic edge detection, but need "a good one" for things like OCR.
I don't find a ton of competitive difference amongst the "good ones" especially not reflected in the pricing and marketing. IMO there are pretty much just "non-AI" algorithms, "basic AI" that are getting more interesting on-device for some use cases, and "good AI" or what we'd call "frontier models" - which again, not seeing a ton of differentiation on that layer.
overgard 18 hours ago [-]
I concur that there isn't that much difference between the good ones. I'm not really trying to push the limits of what they can do or anything, but I'd say on average I give it tasks about the size of something that would take me a half day to do by hand, and honestly, other than minor quirks in coding style and word choices, they've all basically been "good enough" for what I want. When I tried fable it just wasn't for me because it was basically very expensive overkill. I usually stay on the older models as long as I can just because they're predictable and I care a lot more about predictability and cost than being on the super cutting edge or whatever. (Honestly I'd kind of rather the enthusiasts test it first for the various "new model deleted my database" bugs that seem to pop up.
cyanregiment 11 hours ago [-]
Yeah definitely good enough for work. I think people only get into trouble going “agentic” and trying to automate je ne sais quoi
LoganDark 19 hours ago [-]
I feel like nearly anything done by a "Lo" is better done myself. I use "Hi" for pushing the limits of my expertise.
cyanregiment 18 hours ago [-]
A good use case for me is automating invoice notes that are all very similar, but change based on details of client and work. Mistral 7b does it fine, instantly.
UX-wise, I don't predict out "ghost text" like Google has been doing, I just preload the input value with the prediction. You can edit as you want. Saves a ton of time.
jurgenburgen 5 hours ago [-]
There’s a heuristic I’ve been using to pick which model to use. The provider is already chosen by my org so what I do is activate the model picker in my harness, select the most expensive model’s latest version and set it on the highest effort setting.
This has worked quite well for me and saves a lot of time.
mrkeen 11 hours ago [-]
A programmer is someone who declares "I understand the problem so well that I can teach a computer to solve the problem, even for examples I have never seen before".
If it's too hard for a programmer to reason about, I wouldn't trust the programmer sitting next to him to automate the problem away.
brunaxLorax 14 hours ago [-]
It's hard to keep track of all those launches, I agree. I would separate 2 usages of LLMs:
- LLM is integrated into a workflow/app for a classification or whatever job: scoring outputs (evals, KPIs) and choosing a fit between quality and cost is enough
- LLM is used as a tool you interact with (like coding or research): here is where you have to know your brushes. As humans we get used very quickly by the model personality, the format of output, the way it "thinks" and interacts.
danielmarkbruce 21 hours ago [-]
You don't just try every model that comes along... and for the ones you do, you have eval pipelines set up. It's engineering, you have to test stuff works.
Now, there are thousands of use cases for lots of the fontier models. I don't use them for everything, so I don't test them except on the stuff I'm using them for.
overgard 17 hours ago [-]
Well, first off, I don't need to test that stuff works, the labs need to do that! I can afford to wait a couple days to get an impression on how people like a model before I switch. I don't want to be a beta tester.
But honestly, even if I went with your idea of picking a small number of models to eval, I wouldn't even know how to choose that subset other than.. what's hottest in the hype cycle? The benchmarks are useless and gamed. There's not a lot of transparency in how they're trained, so there's no way for me to go like "Well, I care about how much graphics programming it's been trained on, and I can see X ingested Y amount of training data on that.." It's basically a stab in the dark of very frothy people saying "X CHANGES EVERYTHING!" with no particular details other than a bunch of guys on youtube being like "I gave it a really lame prompt and it made me this game that absolutely nobody would ever want to play!"
danielmarkbruce 1 hours ago [-]
You are completely wrong and will never make a good product/service with that attitude. You need to test the models work for your use case.
Knowing how much of type X data a model was trained on isn't going to help you one bit. Various phases of training can effectively wipe out the training from earlier steps if not done well. All that matters is does the model do what you want it to do. In practice almost everything in AI/ML is empirical.
overgard 23 minutes ago [-]
Mate, I have 20+ years of experience writing software, I'm not really worried. Frankly if I never used a model ever again I would still write software just fine. These things are conveniences to me, not load bearing.
daeken 19 hours ago [-]
I just ask our fleet maintenance agent to configure the variants of the new model and then run the bakeoff. In a few hours, I get a doc containing exactly how it performed on my real tasks, against the other models we've benchmarked.
We don't do any "prompt engineering" to make specific models work better or worse, so this gives us a direct comparison that lets us objectively compare how they'll work in practice for different types of tasks.
Good enough to have guided multiple model swaps and measured increases in productivity.
mountainriver 24 hours ago [-]
You do still need to know though, different prompts have different affects on them.
zackify 1 days ago [-]
I just use whatever's cheapest for personal things. Dsv4 flash and got 5.6 Luna are great for the price and I'll use the openrouter benchmark view to compare the overall score.
bobmcnamara 24 hours ago [-]
Found the drywaller
cyanregiment 20 hours ago [-]
My landlord just muds over cob webs and open space. They're over 100 years old so they legally count as part of the building's foundation.
So is he, and so does he. He looks like the Tales From The Crypt guy I have no idea how he's still doing manual labor.
stingraycharles 21 hours ago [-]
Just don’t switch models every week and you’ll be fine. Especially in production.
overgard 21 minutes ago [-]
I definitely wouldn't use a model for anything in production!
inigyou 23 hours ago [-]
I'm not surprised that people who use LLMs all day for everything get a feel for the different ones.
TZubiri 6 hours ago [-]
I'm all for dropping priority on vendor neutrality and tightly coupling with vendors.
But model aggregators have a layman smell to me, some laymen think model A talking to model B is materially different than model A talking to model A with another context, also laymen tend to see 'learning AI' as trying out new products and keeping up with product relases.
I think it's less like having a toolbox of screwdrivers, hammers and a wrench, and more like having 30 brands of shovels because there's been a wave of shovel commercials.
iLoveOncall 6 hours ago [-]
Yeah I'm growing increasingly tired of those stupid metaphors about AI.
LLMs are not like screwdrivers, IDEs, or any other tool. They're indeterministic.
I keep seeing everyone make wilder and wilder metaphors which just don't work at all, as a sort of "argument" supporting whatever they say, with 0 actual proof.
paradoxyl 6 hours ago [-]
AIs are like (insert clickbait keyword). You just have to (insert clickbait call to action).
dweez 1 days ago [-]
I spent a lot of time researching LLM routing last year and also came to the conclusion that it's generally not worth the effort. It's too hard to understand the difficulty of a query a priori.
One specific challenge I was seeing is that difficulty depends a lot on what information is retrievable by the agent. Consider the question "what is the 5-state busy beaver number?" (https://en.wikipedia.org/wiki/Busy_beaver). In 2023 this would be a Mythos-tier research problem, but a solution was proved in 2024 so today any minimally intelligent model with a web search tool can just fetch the answer. You don't know which queries will be basic summarization and which will be deep reasoning until you get going.
all2 23 hours ago [-]
I think routing should be pushed 'down the stack' so to speak.
What I mean is that most tasks can be recursively fragmented into smaller tasks, and once you've hit suitable leaf nodes -- where the task is very granular -- you can begin to deterministically show which models perform better or worse for that specific task. Then, when your agent is running a workflow, you may use various models for different steps in a workflow. For example, some models may excel at exploration, some at determining a good architectural fit for an implementation, some at actually writing the implementation, and so on.
But you don't know until you define your 'work' taxonomy, and still further, you won't know until you have a statistically significant number of runs on a given chunk of work. Once you have that, though, you can hone in on models that excel at one specific task or another and prefer those the majority of the time (say ~80%) and hold back the remainder work as a 'test corpus' just in case a different or new model does even better.
This is something I've kept in the back of my head as I've been working through my agent harness primitives -- specifically enabling different models per chunk of work.
dofm 1 days ago [-]
You also have problems with short prompts whose results depend highly on the understanding of nuance. The router is going to have to mostly solve the prompt to decide which model to send it to.
What you actually want is a model that can conclude either "I know the answer to this with confidence" and answer, or "I think I don't know the answer to this, I should ask another model and I know which one". But I don't think LLMs can really bring their uncertainty to the surface in that way yet? Their internal confidence can be measured and returned, so you could probably front a more powerful model with a knowledgeable assistant, but they can't consciously mark their own homework?
zbentley 1 days ago [-]
> Their internal confidence can be measured and returned
It can? I was under the impression that confidence was either self-reported by the LLM or assessed by having another model interpret the output response. If there's a confidence score at the level of the actual model math, that's news to me.
dofm 24 hours ago [-]
You certainly can't ask an LLM for its confidence level because it will generate that answer as it sees fit.
My understanding was that models can be post-trained to assess their own confidence on short prompts within a level of accuracy (there was an article about this a few weeks ago that I can't find), but can't use it to reason, and that research has shown that internally they effectively have a measurable sense of truth but can't surface it:
I mangled what I was trying to say but the point I guess is, it is effectively in there, and researchers can see it and therefore score it, but it is not something the LLM can use.
The problem seems to me (layman's understanding at best) that the LLM is inherently confident within the words of its answer, because of what the model is trained to do and how it is trained. So you can never get an accurate "I don't know this" from a model while it is answering; it is bullshitting.
(One of the things that is most interesting to me at the moment is asking a small local LLM what it knows about a topic and then testing it. It can have no idea of what it wasn't trained on, but any question about what it knows is seemingly going to trigger it to work through all the summary word associations it did find in training, so it can sort of summarise its knowledge that way, with some likelihood of success, without any sense of introspection)
Ahh you found the link I couldn't find with a search.
Yes — they claim that it is likely to be able to assess confidence in whether the statement it just made is correct. But it isn't going to be capable of that while it is answering.
TZubiri 6 hours ago [-]
Logprobs
TZubiri 6 hours ago [-]
How does one spend a lot of time researching model routing? Did you evaluate different routing providers? Maybe you could build a rorouterter that lets you easily switch to any router, ideally if there's a vuln in any of them you can get infected even if you don't use it.
JarJarBeatU 23 hours ago [-]
I mean can't you just directly test some variations of the query against many models at once and pick the cheapest model that hits your accuracy goal? If it's a one time run then ofc this is all pointless, but for ongoing tasks it makes sense. This seems more logical to me than making another AI model of some sorts intuit the right LLM for the job.
velcrovan 1 days ago [-]
Ironically, my confidence that a human had at least an active part in writing/editing this article went up because of this train wreck of a sentence:
> "A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it."
brunaxLorax 14 hours ago [-]
a French human doing its best to sound punchy in English :)
hhh 1 days ago [-]
what’s wrong with the sentence? reads fine to me
inigyou 23 hours ago [-]
Weird tense combination. I'm not sure if it's officially wrong but it feels weird to change tense mid-sentence after so many words. "X will do ABCDEFGHIJKLMNOPQRSTUVWXY and does Z."
I thought it could be a typo for "X will do ... and do Z" but only when I went to write why it's weird, I realised it doesn't have to be a typo.
It doesn't even matter here. It would mean the same thing either way.
1bpp 1 days ago [-]
'keeps' is the wrong tense and could be 'will keep'
thehamkercat 1 days ago [-]
i think it should be "by adding stickiness to the initially chosen provider of that model"
jeremyjh 24 hours ago [-]
I agree with one distinction - coding agent workflows can use defined subagent roles that are pinned to specific models and I have found this very effective. The orchestrator is building all the context to make these assignments - it’s not a dumb router. Using Minimax M3 for exploration and librarian tasks for example is fast and cheap - my $10 plan lasts all month and saves a lot of tokens for my main coding plan.
brunaxLorax 14 hours ago [-]
Yes. orchestrator > smart router
try-working 1 days ago [-]
I recently wrote about first principles of model routing that I've learnt building a model router.
The model pool should be kept small, and models in the pool should be clearly differentiated. For example, one large frontier model for quality, one small, fast and cheap model like DeepSeek V4 Flash for routing work.
These two principles by themselves solve the issues with caching, with routing decision making. I routinely hit >99% cache while routing between GPT 5.4 and DeepSeek.
I think this should probably be scoped to 'generic router systems that don't understand query context' are not useful. We have had lots of good results with routers that understand the context of the types of workloads they process and can route requests to the most efficient models.
randomblock1 1 days ago [-]
> Complexity cannot be deduced from the prompt alone. Let’s take an example: “evaluate the tests for the repo $GIT_REPO and improve them” can be a very simple task if you mention a personal website written in plain HTML5; or an incredibly complex task if you target the Linux kernel repo.
I don't think this a good example. The first step would be reading the documentation, reading an overview of the tests, then executing commands to run the tests. A cheap model could do that. After that, though, the router will have to figure out how complex the tests are, which is the hard part, but I can't imagine it's that hard to determine a bunch of C code is tougher than some HTML from looking at it. Unless they want to select one model at start and never change it, because that's pretty clearly not the right way to go about routing.
Regardless I agree that routers usually aren't worthwhile, at least in the form of something that's meant to be universal. It's probably more efficient to just change something in the repo code, whether that's skills or instructions or something else. Benefit of that being it's persistent, portable, and more well tuned than guessing complexity on every turn.
brunaxLorax 14 hours ago [-]
Sounds like an orchestrator for me ratter than a router
kaycey2022 17 hours ago [-]
It seems apparent to me that task complexity can’t be determined by prompt alone. How? A prompt is just a simple rambling. An agent will go through many many tool calls and steering just to arrive at the right approach.
A serious router therefore needs to build up a dataset of how different models responded, end to end, to different tasks on different contexts. I won’t comment on whether the current frontier models can reliably judge these outputs, but I am sceptical about that. And moreover look at the state of evils! They are gamed to hell and keep losing credibility.
A more difficult problem for router builders is that they are working on an opaque system behind an external API. How can you reliably guarantee model behaviour when model behaviour has been shown to deteriorate under arbitrary conditions that have nothing to do with the task given? So much investment only to be an AI company that can get rug pulled by the real AI companies at any given time.
I would think the only people who can come up with good routers for a collection of models are the inference providers themselves. Because theoretically they have full control of how their models are served. And inference is not zero cost or cheap for them either. And going by OpenAI’s experience routing is not an easy problem for them to solve either. And they don’t have the incentive to route you to cheaper models and reduce costs for customers at the same time. Routing objective for them is to increase their own profits.
mmargenot 1 days ago [-]
I find this routing problem to be opaque and I’m generally skeptical that the label people are trying to predict is meaningful.
If you really need more discrimination of the complexity of an input to get an efficient response, sft or rl tuning something for your harness would be more effective.
firasd 20 hours ago [-]
The 'personality' of the model matters a lot even for stuff like Code Gen. If you're building something with Fable and then get routed to Opus cause the router decided your task is not 'demanding enough' it's like: great, now I have to read complex prose like "That's the thing, and the thing is the point:..." Opus-isms when I just wanted to make a shopping cart
brunaxLorax 14 hours ago [-]
Agree
luciana1u 19 hours ago [-]
every LLM router eventually reduces to a config file that says "use the good one" and nobody can define what good means
atlex2 23 hours ago [-]
What was the architecture of your router? If it was based on GRPO/RL, it would be interesting to hear why your router performance capped.
I think the truth is that it's not an efficient cost cutting method. Your router has to be at least as 'smart' as all the but the smartest of your models (models do poorly when asked 'is this a task you're well suited to'), and that means you're caching multiple prompt histories including kv-filling/prefix caching on your expensive router model. Most of the time, not super great for savings.
brunaxLorax 14 hours ago [-]
We tried different things: heuristic, TF-IDF and LLMs. I voluntarily didn't talk about the tech because it doesn't fix the problems mentioned by switching models on the fly.
scionaura 24 hours ago [-]
Insider take: routing will not be a (successful, durable) thing, at least not externally to model providers.
The labs are incentivized to solve this problem themselves, since they’re competing on a 2D cost-intelligence frontier. If they can reduce cost without harming intelligence they will do that and pass on (some of) the cost reduction to the user. There are nicer solutions available to them because they can cut into lower levels of abstraction.
E.g. you should consider speculative decoding to be one (very conservative) form of routing and note that you can’t implement that for the labs from the outside.
CuriouslyC 23 hours ago [-]
This arbitrage isn't durable either though, as it's just a race to the bottom with open research, and the long term destination of everything is custom models for different tasks, since it's going to become increasingly apparent that some areas of knowledge have anti-synergy as we push up the diminishing return curve.
brunaxLorax 14 hours ago [-]
100% agree. MoE is the perfect example: reducing FLOPs while keeping intelligence
rrvsh 16 hours ago [-]
Didn't this slightly contradict itself? The conclusion I drew based on the problem statement was that the best way was to have a model router handle the root prompt and stick to a model for the rest of the session - engineers still have the discretion to handoff to a smaller or larger model when needs arise.
0xDEAFBEAD 24 hours ago [-]
From a big-O notation perspective, if you're serving N queries, working the kinks out of your routing system costs perhaps O(logN) in developer time, whereas LLM provider savings grow as O(N).
In other words, the more queries you're serving, the more worthwhile it looks to figure out a viable method of model routing.
blackcat201 20 hours ago [-]
I think the author need to rethink out of the box what's LLM routers in a traditional sense ( input in, route, output ) and move to think how a router would work in agentic workflow. See cognition Devin Fusion design.
To add: I doubt frontier labs will build routers - they are not financially incentivized to optimize token usage (though in the short term they may be incentivized by constrained GPU capacity to reduce load).
brunaxLorax 14 hours ago [-]
All inference providers (labs and neoclouds like TogetherAI or Fireworks) are incentivized to be efficient to be more competitive. For example MoE reduces compute without reducing output quality. I would not be surprised if they end up implementing some kind of internal routing at some point.
owenthejumper 1 days ago [-]
Routing belongs on the client side
hrpnk 1 days ago [-]
Even the simplest router that differentiates models between plan & execute will ensure that this is consistently followed. Folks have too high FOMO to choose themselves.
KoleSeise1277 20 hours ago [-]
Classifying complexity from the prompt alone was always the weak point. Most of what makes a task hard shows up after the first tool call.
htrp 24 hours ago [-]
>Our router was classifying each request into one of four different tiers of complexity: simple, standard, complex and reasoning.
Seems like a naive classification model lacking context?
brunaxLorax 14 hours ago [-]
Could be. In that case many routers are concerned as they all have more or less the same category buckets.
maxrev17 1 days ago [-]
Yeah routers suck, been doing this myself for the best part of a year now and it’s really difficult to make it behave. A good model for your task is the best bet.
robertclaus 1 days ago [-]
I think fail-over for a production critical service is an equally important responsibility of the router.
ramraj07 1 days ago [-]
You cant fail over to a different model though. Even failing over to a different provider isn't always as simple as set it and forget it. For example bedrock doesn't support the tool search tool feature from anthropic.
brunaxLorax 14 hours ago [-]
I didn't know that! I thought that all models had the same features no matter the provider
jml78 1 days ago [-]
They beta support for it now I believe
brunaxLorax 14 hours ago [-]
Yes, providers have downtime and retry/fallback solves this problem.
seizethecheese 1 days ago [-]
Their router classified prompts into difficulty buckets. This obviously won't work. Consider a senior developer routing work based only on the task description. Clearly you need to dig a bit deeper into the task.
Saying routers don't work is sort of like saying serverless doesn't work. It depends on when and how!
One routing implementation that recently launched here is interesting (https://news.ycombinator.com/item?id=49099143). It routes based on the models' initial trajectories. This is like having multiple developers get started, seeing what they're doing, then pulling all but one off the project. It should work, but doesn't seem ideal!
I've gotten routing working well for typical chatbot prompts in http://pellmell.ai. This is fine because prompts are easy to classify into category buckets (for example: legal, medical, general knowledge, code). And models definitely have strengths and weaknesses. You want Gemini to answer General Knowledge and you want Claude to answer coding.
brunaxLorax 14 hours ago [-]
I didn't know about Tokenless, the approach seems really innovative, if it works it fixes the "Complexity cannot be deduced from the prompt alone" problem. However you still have the other hidden costs: cache, breaking behavior consistency, and unpredictability.
Your point is interesting, you say that task specificity is easier to classify than task complexity, which I agree - I didn't mention it but we had task specificity routing too :). My opinion is that in many cases task specificity calls are easy to distinguish at build time, and therefore you can isolate them and attach the right model/settings beforehand, so you have less need for a smart routing on the fly.
coffinbirth 1 days ago [-]
Some routing services not just route to different LLMs, they also handle all the legal issues (GDPR compliance, ISO certification, guaranteed Zero-Data-Retention, domestic data processing/European based clouds, etc.). In regulated industries, these things matter a lot, especially when processing of sensitive data is involved.
inigyou 23 hours ago [-]
By "guaranteed" they mean "we promise really hard" right? Data retention has so much profit potential that any provider would have to be completely stupid not to retain data, even after promising they didn't.
brunaxLorax 14 hours ago [-]
If your router has GDPR/ZDR/EU compliant, it doesn't make your provider compliant. You're just adding a middleman.
Now that even the smaller models from labs (Luna, deepseek v4 flash) are getting powerful, I think the orchestrator pattern of a smart model coordinating smaller models for work will end up being the way to go.
brunaxLorax 14 hours ago [-]
Agree. The orchestrator pattern (big model managing smaller ones) works already well
joshowens 17 hours ago [-]
[flagged]
scotttaylor 1 days ago [-]
[flagged]
20 hours ago [-]
fibuladev 23 hours ago [-]
[dead]
jing09928 20 hours ago [-]
[flagged]
aegisora_ai 20 hours ago [-]
[flagged]
youre-wrong3 1 days ago [-]
[dead]
Rendered at 21:06:57 GMT+0000 (Coordinated Universal Time) with Vercel.
I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially good at is more or less just throwing spaghetti against the wall, except the spaghetti is potentially very expensive and might insert subtle issues into your code base.
The better I get at recognizing what each model is good/bad at, the more I'm glad we're taking the time to choose specific models for specific prompts -- and the more I wouldn't trust a generic router to efficiently route for me.
Maybe Gemini or ChatGPT is a "Hi" while running a local Mistral 7b is a "Lo" - sometimes you want lo-res technologies. Faster, lighter, does the job.
Find the same with image, video, vision, and so-on.
Probably get away with pretty weak vision models for some basic edge detection, but need "a good one" for things like OCR.
I don't find a ton of competitive difference amongst the "good ones" especially not reflected in the pricing and marketing. IMO there are pretty much just "non-AI" algorithms, "basic AI" that are getting more interesting on-device for some use cases, and "good AI" or what we'd call "frontier models" - which again, not seeing a ton of differentiation on that layer.
UX-wise, I don't predict out "ghost text" like Google has been doing, I just preload the input value with the prediction. You can edit as you want. Saves a ton of time.
This has worked quite well for me and saves a lot of time.
If it's too hard for a programmer to reason about, I wouldn't trust the programmer sitting next to him to automate the problem away.
Now, there are thousands of use cases for lots of the fontier models. I don't use them for everything, so I don't test them except on the stuff I'm using them for.
But honestly, even if I went with your idea of picking a small number of models to eval, I wouldn't even know how to choose that subset other than.. what's hottest in the hype cycle? The benchmarks are useless and gamed. There's not a lot of transparency in how they're trained, so there's no way for me to go like "Well, I care about how much graphics programming it's been trained on, and I can see X ingested Y amount of training data on that.." It's basically a stab in the dark of very frothy people saying "X CHANGES EVERYTHING!" with no particular details other than a bunch of guys on youtube being like "I gave it a really lame prompt and it made me this game that absolutely nobody would ever want to play!"
Knowing how much of type X data a model was trained on isn't going to help you one bit. Various phases of training can effectively wipe out the training from earlier steps if not done well. All that matters is does the model do what you want it to do. In practice almost everything in AI/ML is empirical.
We don't do any "prompt engineering" to make specific models work better or worse, so this gives us a direct comparison that lets us objectively compare how they'll work in practice for different types of tasks.
Good enough to have guided multiple model swaps and measured increases in productivity.
So is he, and so does he. He looks like the Tales From The Crypt guy I have no idea how he's still doing manual labor.
But model aggregators have a layman smell to me, some laymen think model A talking to model B is materially different than model A talking to model A with another context, also laymen tend to see 'learning AI' as trying out new products and keeping up with product relases.
I think it's less like having a toolbox of screwdrivers, hammers and a wrench, and more like having 30 brands of shovels because there's been a wave of shovel commercials.
LLMs are not like screwdrivers, IDEs, or any other tool. They're indeterministic.
I keep seeing everyone make wilder and wilder metaphors which just don't work at all, as a sort of "argument" supporting whatever they say, with 0 actual proof.
One specific challenge I was seeing is that difficulty depends a lot on what information is retrievable by the agent. Consider the question "what is the 5-state busy beaver number?" (https://en.wikipedia.org/wiki/Busy_beaver). In 2023 this would be a Mythos-tier research problem, but a solution was proved in 2024 so today any minimally intelligent model with a web search tool can just fetch the answer. You don't know which queries will be basic summarization and which will be deep reasoning until you get going.
What I mean is that most tasks can be recursively fragmented into smaller tasks, and once you've hit suitable leaf nodes -- where the task is very granular -- you can begin to deterministically show which models perform better or worse for that specific task. Then, when your agent is running a workflow, you may use various models for different steps in a workflow. For example, some models may excel at exploration, some at determining a good architectural fit for an implementation, some at actually writing the implementation, and so on.
But you don't know until you define your 'work' taxonomy, and still further, you won't know until you have a statistically significant number of runs on a given chunk of work. Once you have that, though, you can hone in on models that excel at one specific task or another and prefer those the majority of the time (say ~80%) and hold back the remainder work as a 'test corpus' just in case a different or new model does even better.
This is something I've kept in the back of my head as I've been working through my agent harness primitives -- specifically enabling different models per chunk of work.
What you actually want is a model that can conclude either "I know the answer to this with confidence" and answer, or "I think I don't know the answer to this, I should ask another model and I know which one". But I don't think LLMs can really bring their uncertainty to the surface in that way yet? Their internal confidence can be measured and returned, so you could probably front a more powerful model with a knowledgeable assistant, but they can't consciously mark their own homework?
It can? I was under the impression that confidence was either self-reported by the LLM or assessed by having another model interpret the output response. If there's a confidence score at the level of the actual model math, that's news to me.
My understanding was that models can be post-trained to assess their own confidence on short prompts within a level of accuracy (there was an article about this a few weeks ago that I can't find), but can't use it to reason, and that research has shown that internally they effectively have a measurable sense of truth but can't surface it:
https://arxiv.org/abs/2410.02707
I mangled what I was trying to say but the point I guess is, it is effectively in there, and researchers can see it and therefore score it, but it is not something the LLM can use.
The problem seems to me (layman's understanding at best) that the LLM is inherently confident within the words of its answer, because of what the model is trained to do and how it is trained. So you can never get an accurate "I don't know this" from a model while it is answering; it is bullshitting.
(One of the things that is most interesting to me at the moment is asking a small local LLM what it knows about a topic and then testing it. It can have no idea of what it wasn't trained on, but any question about what it knows is seemingly going to trigger it to work through all the summary word associations it did find in training, so it can sort of summarise its knowledge that way, with some likelihood of success, without any sense of introspection)
Yes — they claim that it is likely to be able to assess confidence in whether the statement it just made is correct. But it isn't going to be capable of that while it is answering.
> "A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it."
I thought it could be a typo for "X will do ... and do Z" but only when I went to write why it's weird, I realised it doesn't have to be a typo.
It doesn't even matter here. It would mean the same thing either way.
The model pool should be kept small, and models in the pool should be clearly differentiated. For example, one large frontier model for quality, one small, fast and cheap model like DeepSeek V4 Flash for routing work.
These two principles by themselves solve the issues with caching, with routing decision making. I routinely hit >99% cache while routing between GPT 5.4 and DeepSeek.
https://try.works/first-principles-of-model-routing
I don't think this a good example. The first step would be reading the documentation, reading an overview of the tests, then executing commands to run the tests. A cheap model could do that. After that, though, the router will have to figure out how complex the tests are, which is the hard part, but I can't imagine it's that hard to determine a bunch of C code is tougher than some HTML from looking at it. Unless they want to select one model at start and never change it, because that's pretty clearly not the right way to go about routing.
Regardless I agree that routers usually aren't worthwhile, at least in the form of something that's meant to be universal. It's probably more efficient to just change something in the repo code, whether that's skills or instructions or something else. Benefit of that being it's persistent, portable, and more well tuned than guessing complexity on every turn.
A serious router therefore needs to build up a dataset of how different models responded, end to end, to different tasks on different contexts. I won’t comment on whether the current frontier models can reliably judge these outputs, but I am sceptical about that. And moreover look at the state of evils! They are gamed to hell and keep losing credibility.
A more difficult problem for router builders is that they are working on an opaque system behind an external API. How can you reliably guarantee model behaviour when model behaviour has been shown to deteriorate under arbitrary conditions that have nothing to do with the task given? So much investment only to be an AI company that can get rug pulled by the real AI companies at any given time.
I would think the only people who can come up with good routers for a collection of models are the inference providers themselves. Because theoretically they have full control of how their models are served. And inference is not zero cost or cheap for them either. And going by OpenAI’s experience routing is not an easy problem for them to solve either. And they don’t have the incentive to route you to cheaper models and reduce costs for customers at the same time. Routing objective for them is to increase their own profits.
If you really need more discrimination of the complexity of an input to get an efficient response, sft or rl tuning something for your harness would be more effective.
I think the truth is that it's not an efficient cost cutting method. Your router has to be at least as 'smart' as all the but the smartest of your models (models do poorly when asked 'is this a task you're well suited to'), and that means you're caching multiple prompt histories including kv-filling/prefix caching on your expensive router model. Most of the time, not super great for savings.
The labs are incentivized to solve this problem themselves, since they’re competing on a 2D cost-intelligence frontier. If they can reduce cost without harming intelligence they will do that and pass on (some of) the cost reduction to the user. There are nicer solutions available to them because they can cut into lower levels of abstraction.
E.g. you should consider speculative decoding to be one (very conservative) form of routing and note that you can’t implement that for the labs from the outside.
In other words, the more queries you're serving, the more worthwhile it looks to figure out a viable method of model routing.
Seems like a naive classification model lacking context?
Saying routers don't work is sort of like saying serverless doesn't work. It depends on when and how!
One routing implementation that recently launched here is interesting (https://news.ycombinator.com/item?id=49099143). It routes based on the models' initial trajectories. This is like having multiple developers get started, seeing what they're doing, then pulling all but one off the project. It should work, but doesn't seem ideal!
I've gotten routing working well for typical chatbot prompts in http://pellmell.ai. This is fine because prompts are easy to classify into category buckets (for example: legal, medical, general knowledge, code). And models definitely have strengths and weaknesses. You want Gemini to answer General Knowledge and you want Claude to answer coding.
Your point is interesting, you say that task specificity is easier to classify than task complexity, which I agree - I didn't mention it but we had task specificity routing too :). My opinion is that in many cases task specificity calls are easy to distinguish at build time, and therefore you can isolate them and attach the right model/settings beforehand, so you have less need for a smart routing on the fly.