About six years ago we built a ticket classification/routing model and pipeline [1]. The initial version was trained using SVMs, it took a couple of minutes to train, and the ticket classification was just a few milliseconds. It's so fascinating to watch how the field moved over to LLMs, and then back to faster/cheaper classifiers again.
NOTE: the work was actually accepted at the International Conference on Software Engineering (ICSE) in 2021.
jval43 15 hours ago [-]
SVMs in big data were already well liked back in 2010 as they could be scaled via map-reduce.
An issue in industry was that companies would not allow training on their data across customers. Understandably so, as SVM dimensions are readily interpretable given some thought. General purpose datasets were ill-suited too depending on the task.
So you'd start from scratch every time, annotate features like crazy, and often didn't have enough training data overall.
LLMs "solved" that by just ignoring the previous rules and training on everything, which lets the necessary world knowledge seep back in.
I'm sure we'll see many combinations of LLMs and regular machine intelligence in the near future.
miki123211 12 hours ago [-]
LLMs, and other transformer models before them like BERT for example, have the benefit of "transfer learning."
If you want to make a ticket classification model, you don't need to train across many customers doing ticket classification, who might not want you to train on their data and whose data is impossible to obtain unless you already are a large company. Instead, you can train on a completely different task which forces the model to learn the syntactic and semantic structure of the English language — say predicting missing words in Wikipedia articles, which you can't do successfully without a good understanding of English — and then fine-tune that model on your specific task with much less data.
Because the initial task is so general, you can have one company do the expensive pretraining, share the base model, and then have everybody else just do the finetuning on their (small) proprietary dataset.
LLMs and models like Jev go one step further, letting you use them as classifiers with no training at all, with nothing but rules in the prompt. And with a few tricks, you can indeed use them to get pretty high-quality labels, which you can then distill into a fine-tuning dataset for a much smaller model.
thesz 11 hours ago [-]
> say predicting missing words in Wikipedia articles, which you can't do successfully without a good understanding of English
This is word2vec model, one of - predict a word from context. The other one is to predict surrounding words (context) from a word. It did a good job when I used it, words' classes were impeccable for most practical purposes.
Word2vec is a two layer perceptron.
mrbungie 11 hours ago [-]
¿Por qué no los dos? Big skill issue, you can even route known problems to a trained in-house simple classifier and low-confidence problems to Jev-like "zero shot classifiers" or LLMs.
I feel like people in the field forgot everything.
ctxc 7 hours ago [-]
I'm sorry, can you restate - are you saying in house models are good enough even when there's no relevant training data?
asgraham 6 hours ago [-]
They seem to be saying: route data that matches training data to the trained classifier, and route the rest to Jev.
cogogo 10 hours ago [-]
It will only get worse. Non-tech people will be (or are) very suspicious of anything trained in house. The bar to spend time/money on anything domain specific will only get higher.
pavon 5 hours ago [-]
Training in-house has never been a skill issue, it is a budget issue of collecting and labeling enough training data.
hilariously 7 hours ago [-]
They didn't forget, they saw that nobody was paying for anything else while the rest of the larpers showed up and literally do not know anything about ML and just think "lemme paste some chadbbd on this"
smcg 6 hours ago [-]
exactly. they wouldn't know a SVM from a SVG.
Aurornis 8 hours ago [-]
> LLMs "solved" that by just ignoring the previous rules and training on everything, which lets the necessary world knowledge seep back in.
True, but could be taken the wrong way. LLMs were also not able to train on that private customer data. They trained on so much general public data that they became capable enough to operate generally even on private data they had not been trained on.
That was the big breakthrough. I know that’s what you were trying to say, but on sites like this there are a lot of people who will see a comment like that and assume that LLMs like Sonnet were fed all of the private customer data that was withheld from training other models.
smcg 6 hours ago [-]
public data and stolen non-licensed data*
abhgh 12 hours ago [-]
Agree to all of this but I would make a minor correction: the interpretability is true under certain conditions, e.g., you use something like n-grams and train an SVM with a linear kernel. You could change the representation back then too, e.g., LDA (topic models), Latent Semantic Indexing, and of course, you could change the SVMs kernel as well, e.g., use a sequence kernel [1]. Some of these were much less interpretable. But the common pipeline back then was indeed n-grams + Linear SVM. Today it is easy to modernize the pipeline by just changing the representation step to use an embedder like ModernBERT. I have a "one page" classifier if you want to try it out - [2].
[1] I went down the rabbit hole of using sequence kernels quite a bit back then, see https://arxiv.org/abs/1910.13800, but the timing was terrible. RNNs and variants like LSTMs were beginning to become popular, and everyone (including me) switched.
And you can now get that world knowledge out of llms and generate synthetic data to train your svm and fine tune on your own dataset!
flatline 13 hours ago [-]
I’ve hand-labeled so many freaking datasets over the years for supervised learning tasks, if only to build out a PoC. Would 100% hand this off to an LLM and just fine tune the results. I definitely do not want an LLM in the loop at inference time for anything I could build a simple classifier for.
IanCal 12 hours ago [-]
Llm labelling is so incredibly useful, as is transforming labelled data to other valid but different forms (like where you might train on sentences but also randomly replace days of the week with others).
You can also do stuff like setup active learning but have the labeller be an llm as well to make things more efficient.
Definitely worth it for PoC level things if not full ones.
Edit - also reviewing, so it can run training, test, review failure cases and investigate itself. You can leave them to plough through a few iterations where the early part is the classic time you’d have a few obviously bad sets of results.
benrutter 15 hours ago [-]
Yeah, its pretty wild! I think one of the benefits, but also potential risks, with LLMs, is you can just point them at anything without a lot of expertise.
Building an SVM isn't a tricky job, but you probably need a data scientist involved, and to think carefully about a labeled dataset.
I think we're seeing the modelling equivalent of something like electron apps. Where they waste a tonne of resources, but make an easier developement experience.
Will be interesting to see how much or little this changes with LLM costs.
Foobar8568 13 hours ago [-]
I have seen Phd in stats validating models with data the models have been trained on. And I don’t live in a country with dubious universities.
So between llm and pseudo experts, at this stage, I prefer to go with a LLM.
IshKebab 14 hours ago [-]
> Building an SVM isn't a tricky job, but you probably need a data scientist involved, and to think carefully about a labeled dataset.
Well not anymore... You can just get AI to build it.
benrutter 13 hours ago [-]
True I guess. I think most companies would at least want sign off from someone who's heard of SVMs that it isn't doing something mad.
Just like, LLMs can build web back ends, but I'd want a security professional to give it some kind of thumbs up. Especially if a lot of money rides on it.
IshKebab 2 hours ago [-]
Maybe for security. I dunno if most companies would care for SVMs because they're intrinsically probabilistic.
Also IMO it's a lot easier to skim something and say "yeah that looks like it does what we want" than it is to come up with it from scratch.
armcat 11 hours ago [-]
I've seen a few people comment along the lines of "yes but LLMs dont need lot of data nor competence". So with our classifier "Linnaeus", the data came from JIRA tickets plus additional signals. To get up and running we didn't use a lot of data (a few dozen tickets across approximately 20 classes to begin with - eventually expanded to 120+ classes). There was very little to no manual data labelling - mostly it was devs selecting "gold standard" JIRA tickets for us to ingest. The model was retrained daily, and we had extra signals like thumbs up/down with comments that would be used for re-training.
solidasparagus 14 hours ago [-]
What has changed is that expensive models can be deployed instantly and generate training data until you swap over to training your own cheap models.
Your model only took minutes to train if you don't count the data collection and labelling work
evolabs 9 hours ago [-]
although in our experience, synthetic data != real data. You need to be quite careful about the distribution (or just try to get public real data, or label stuff yourself)
matthorse 13 hours ago [-]
It's pretty interesting how a very broad field of machine learning got reduced to only LLMs for most consumers. Probably it's due to marketing (and of course how powerful these models are), but their lack of capability in many tasks already forces people, even amateurs, to use LLMs to write deterministic code instead. I wonder if the sudden appearance of viral classifiers predicts more common use of smaller ML models as the next step?
miki123211 12 hours ago [-]
non-LLM models just aren't visible to consumers.
Saying that the field got reduced to only LLMs is like saying that the field of combustion engines got reduced to cars. The trucks, combine harvesters and peaker plants are still here, they're just in the background, invisible to people who are not part of the industry.
Banks still use random forests and SVMs, medicine and image processing still use CNNs (and people who take photos with their phone don't even realize how much image processing is being done), TTS, ASR and predictive typing models are in wide use, and there's probably still plenty of BERT in spam classification and such.
TYPE_FASTER 5 hours ago [-]
> non-LLM models just aren't visible to consumers.
Yeah, exactly. Your phone just...does stuff. Bar/QR code scanning is probably using a CNN to minimize resource utilization.
LLMs are like an ice breaker to talk about AI and all the ways it has existed for decades, and what AI really refers to.
autuni 15 hours ago [-]
I think this kind of change will pop up more often.
In software engineering, we learned a long time ago that giant monoliths are not the answer. You want smaller, but specialized units that work together. They perform better at their specific task, are more maintainable, you can just upgrade one part of the system without having to impact the others, ....
I don't know where the focus on building a single model that can do everything comes from.
MaKey 9 hours ago [-]
> In software engineering, we learned a long time ago that giant monoliths are not the answer.
That depends on the question. If the question is "How do I utilize 10,000 software engineers?" then yes, a monolith is not the answer. However, this isn't a problem most companies face. Starting out with microservices is rarely a good idea either as you'll have a hard time getting the boundaries right.
Geezus_42 15 hours ago [-]
I was thinking about this recently as well. Smaller, specialized models that can run on device just make so much more sense to me than these huge generalized models that need huge infrastructure run.
Danox 6 hours ago [-]
So a small purpose built AI Agent instead of that giant AI model i.e. Made by OpenAI, or Google.
So if that’s a part of the future, what’s the purpose of spending billions of dollars on large AI models? Is this just a continuation of the story that there is no moat? For the big model makers?
Why do you need a master of everything? when you can just use a much smaller specialist AI model.
bityard 9 hours ago [-]
The ability to do "everything" is an emergent property of how they are built and trained.
If you want a model that can do X but don't care about Y And Z, intuition says you only have to train the model on data relating to X. However, early LLM research showed that doing this results in a far less X-capable model than one trained on data for Y and Z as well.
ivell 14 hours ago [-]
I am not sure a best practice in one area translates nicely to another. LLM's power comes with it being a monolith.
autuni 14 hours ago [-]
Does it though? Let's say you use an LLM to write code, do you think it benefits from the fact that the same model is capable of translating corporate messaging into internet slang for marketing?
I would assume a smaller model trained specifically to produce code based on natural language input would perform at least just as well, which would already be better given the smaller size / probably lower training resource requirements.
This is of course, as already said, an assumption. But generalizing typically comes at a cost, there is no reason why this wouldn't apply to LLMs, it even applies to humans.
duskdozer 11 hours ago [-]
They need some of that fluff to keep up the illusion of intelligence and sycophancy that gets people to rely on them. Additionally, they need to make sure they don't prune so much that the model size seems manageable on consumer or standard business hardware, otherwise more people may become tempted to self-host a distilled or other semi-open model. The model size is also a marketing point.
heggy 13 hours ago [-]
Isn't this just the Bitter Lesson over and over again? Specialised approaches get outperformed by more compute and a simpler general approach pretty much every time.
try-working 12 hours ago [-]
Bitter lesson is wrong. Large general models are outperformed by small specialized models.
Danox 6 hours ago [-]
Which is the way it is in life. You need a specialist. when you need to see a doctor, an engineer, a builder, a lawyer, etc, etc the world of AI will be the same. You need a specialist, a model that can focus on exactly what you need and not be cluttered up with what you don’t need.
That might mean that spending all that money on that giant master of none model might lose out to the much smaller specialist model, uh-oh…
9rx 11 hours ago [-]
The Bitter Lesson makes no distinction between large general models and small specialized models, assuming they are both trained from data.
The Bitter Lesson is that human-written rule models will always lose out to trained models in the end.
IanCal 12 hours ago [-]
Or would it be worse as it wouldn’t have the broader knowledge to understand the reason you’re coding what you are anyway, understand the logs and iterate?
Coding an exercise app will benefit from a model that can code and understands what exercise is. A car showroom website will benefit from a model that understands what relevant filters would be.
autuni 11 hours ago [-]
Of course there are synergies between different fields. I'm not saying cutting everything off as close as possible to it's purpose must be the best way. But that's a long way from a single generalized model.
ianjbutler 10 hours ago [-]
> not saying cutting everything off as close as possible to it's purpose must be the best way
Nah, let’s stand firm here and not give in so easily. Parent suggests a coding model needs exercise info in case you want to code an exercise model? Absurd, that leads to the know everything monolith to do anything.
Language, reasoning, tool use, and domain knowledge.. in this case the domain is coding. Someone wants an exercise app? Great, so we look that shit up, we don’t bake it in. We don’t want to bake anything we don’t need in.
We don’t even need to explain this completely reasonable and totally normal engineering preference that small is beautiful. it’s the other side that needs to justify wtf exercise has to do with coding or why everything from twitter posts to a working knowledge of 15th century French poetry just might be helpful for disconnected domains, like say serious work in contemporary astrophysics. Back to basics, No one asked for that, No one should accept as given that this is the way anything needs to work
IanCal 10 hours ago [-]
> Parent suggests a coding model needs exercise info in case you want to code an exercise model?
No, the point is having broader knowledge than just "can code" is important for building software for humans to use.
> We don’t even need to explain this completely reasonable and totally normal engineering preference that small is beautiful.
It is, of course it is, as long as that small thing is useful. My argument is not "let's put literally everything in this thing just in case" it's "broader world knowledge is important for building many things".
Also, we don't look at CVs or histories of people and say "hmm, this person could be good at coding for us but I really wish they didn't have such a broad and expansive history".
> Great, so we look that shit up
But what do you have it look up? What is exercise? What is a muscle and what is a human? How is it going to understand those references? Does it have the broad enough understanding to see where the gaps are?
At some level unless all you want is "loop over the array and sum the results into a variable called sales_total" converted into code, you have to have some broader knowledge about things.
And once you start considering that maybe it needs more than simply code, we're not disagreeing we're just discussing where the line should be. Which I've also not placed, because there's no simple statement to make on that.
ianjbutler 9 hours ago [-]
> Also, we don't look at CVs or histories of people and say "hmm, this person could be good at coding for us but I really wish they didn't have such a broad and expansive history".
Damn right we don't say that. And I don't look at a carpenter and wonder if his head is hard enough to drive nails like a hammer either. People aren't tools, and tools aren't people. You've conflated them because they are both capable of doing work. And that's exactly the type of confusion and category error that the perspective I'm arguing against will lead to.
Wanting a multipurpose tool or system is occasionally reasonable. Much more often it is very, very to close misunderstanding the nature of a problem, what a tool is, or what a system should be. The particulars of purpose and mechanism of action matter, which is why a blowtorch is sometimes a wrench but only if you're kind of desperate. It is always a very scandalous thing to speak ill of someone's best new electric banana-phone / dishwasher-piano that they want to sell you. But that just sounds messy to me, I'm not sure it will improve the sound quality or taste of anything involved.
9rx 13 hours ago [-]
> LLM's power comes with it being a monolith.
Does it? All of the latest frontier models that have made technical details publicly available are using MoE; smaller specialized units that work together.
nixon_why69 11 hours ago [-]
Yeah but they're all trained together along with their gate tensor. Evaluating a moderate length prompt pulls in most/all of them.
wat10000 7 hours ago [-]
I see it as analogous to CPUs and specialized coprocessors. You don't want your general-purpose CPU bit-twiddling your USB data lines or decoding ethernet frames or blitting pixels. Specialized hardware does it so much more efficiently. And yet, powerful general-purpose CPUs remain common. Partly that's because there's a lot of complicated stuff being run that just isn't suited for specialized hardware; try making a C++ compiler coprocessor and see how far you get. And partly it's because that powerful general-purpose CPU is always present and can be instantly reconfigured to do anything. You'd rather use a dedicated cryptography coprocessor than implement your crypto on the CPU, but doing it on the CPU allows you to implement new crypto algorithms quickly. What's better: doing AES on a coprocessor on the CPU? Well, that depends on whether you have an AES coprocessor.
LLMs are the powerful general-purpose CPU of the ML world. Using an LLM as a classifier is woefully inefficient compared to using a dedicated classifier. On the other hand, that LLM exists right now and you can hook it up in ten seconds. You could ask: why use an LLM when a dedicated classifier would be so much more efficient? You could also ask: why build a dedicated classifier when a sufficiently capable LLM is available off the shelf? Neither one is the right answer every time.
darylteo 11 hours ago [-]
> single model that can do everything
"AGI"
ghm2180 8 hours ago [-]
So technically once you have training data via an expensive classifier on NL data, there is no reason one cannot reduce the footprint of the OP classifier to a very small one. I imagine they could just automate feature selection in a SVM(cross validate on CPU!) using an evaluator to train an optimal one.
magicalhippo 6 hours ago [-]
> it took a couple of minutes to train
Does that include collecting, curating and preparing the training dataset? Because that's kinda the hard part I find.
tjungblut 10 hours ago [-]
I have replaced a few of my LLM pipelines with gemma4 embedding and a classifier/regression head now. LLMs are actually really good to gather the data and label it with only prompting, but after a while it is just more economic to train your own model.
mynameisash 7 hours ago [-]
On my previous team at work, the data science team was using Open it's API for tagging of some natural language. The DS team spent weeks working on the prompt plus the outline of the pipeline before I was involved. It took maybe eight to 12 hours to run everything, and it was prone to API failures.
When I asked why we don't build some NLP tagging which would be cheaper and run in a fraction of time, the lead told me, "What if we need to use more unstructured data? GPT can handle that without any extra work."
The team has yet to make any other changes to the pipeline, but in the two-ish years since then, they've spent a ton of money and wasted countless hours on delayed processing because of it. Some day, maybe, they'll learn that traditional ML techniques are superior in virtually every way to just throwing data at an LLM to see what sticks.
pletnes 14 hours ago [-]
Interesting! I had the idea to compute embeddings of log lines to look for “black swans” in my logs. A small model running locally would do. Then PCA, SVM - any old school method should work, I guess. Haven’t seen it done but it must be tried by someone somewhere.
wpmaven 9 hours ago [-]
Thanks for sharing, looks amazing. We have examples of this as well internally.
rgavuliak 9 hours ago [-]
It seems you had enough labels to train, that's not always the case.
chaostheory 15 hours ago [-]
The tech industry is not that far off from the fashion industry
elvircrn 13 hours ago [-]
Svaka cast :)
skwon816 11 hours ago [-]
[flagged]
seanjhkwon 11 hours ago [-]
[flagged]
floro 15 hours ago [-]
This is just the AI grift and uneducated hypebros. There is zero reason to use an LLM as a classifier. But AIbros with zero technical knowledge will push for it anyways even if it is 1000x slower and 1000x more expensive to run.
wodenokoto 15 hours ago [-]
You can can't build an SVM-classifier for a ticketing system without domain data. You can ask an LLM to do it, and it will do it very well.
With the LLM you don't even need a data scientist to train the model, you just need someone who can connect to APIs. And the LLM is fine with adding and removing categories. No retraining.
It makes the initial faces incredibly fast. Then it is slow and expensive to run, so once the system settles, it becomes the target for model optimization, e.g, a dedicated, or semi-dedicated model.
floro 5 hours ago [-]
Yeah but here's the thing: Ease of setting up and how low that bar is, this is not the most important factor for most companies. Also, yes, you need data to do this with an LLM. In fact the LLM was trained on that data for you beforehand. This is not an argument because you can also buy a pretrained classifier.
You empathize the set up cost but ignore that the model is in the magnitude of a million x in size and compute cost of a classifier model. That's like comparing a CPU with an IC
Karrot_Kream 13 hours ago [-]
I mean just have a coding agent do the whole process from start to finish. Tell it you want to seed an SVM from LLM runs on ticket data. I can bet that it'll do it all for you without much input from you at all.
gonzalohm 10 hours ago [-]
I think AI is making people forget that code can be used for automation.
I see coworkers using AI for tasks like "visit an API endpoint, download a file, put it in S3".
This is so stupid. Okay if you want to use AI, but use it to write python code to do that, why pay in tokens every time you want to run that task, it makes no sense
Karrot_Kream 3 hours ago [-]
I see this a lot among non-tech folks, who have a pretty thin understanding of the differences between "code" and "an agent". But do you see this among tech folks as well?
Your S3 case tho I think is a bit misplaced. You're right that if all your keys are configured correctly (API key, S3 key, etc) that a script to do that is like 5 lines in Python (once you get your venv going or use uv to setup the project lol), but tell an agent to just do the thing, and it will make sure that your API keys are in place and that you have access to the S3 bucket you're trying to write to and that the whole thing is in a venv, and then just report back that it's done.
Is it wasteful? Sure. For years I used to feel that way when I saw someone using a messaging app to message a friend across the hall in a loud room or when I saw someone taking a phone picture of their computer monitor and sending it. In the former case your packets are literally going from your phone to a central server and then onto someone else's phone in the same room. But, well, convenience wins.
SoftTalker 7 hours ago [-]
If you don't know what "python code" is, you'll just ask the AI to do the task.
gonzalohm 6 hours ago [-]
But there are teams in your org that could write something in 10 minutes
SoftTalker 5 hours ago [-]
Yes, after the biz unit spends a month talking with IT about it, it might take the developer 10 minutes.
eecks 8 hours ago [-]
How do they use it like that? Like a once off action by the agent or is it embedded in a process?
gonzalohm 7 hours ago [-]
No, they literally just ask. "Can you pull this file using this API endpoint and put it in S3"
Probably under the hood Claude creates some temporary python code or something.
But it bothers me, I don't know why. It's like paying a subscription service for the most stupid tasks
vidarh 14 hours ago [-]
I can spin up a classification solution with an LLM in ca. 5 minutes that is likely to do an okay job. To build a custom classifier, I need lots of training data first.
That is the reason.
Once you then have a system that is used enough, it starts making sense to look to replace that with a leaner classifier, but LLMs have a place in vast number of places where we'd previously never use a classifier because it was too much effort.
floro 5 hours ago [-]
All of the replies talk about how easy and fast it is to set up. That's literally the ONLY reason you would do it with LLMs. If you consider any other factor, it makes zero sense.
darkwater 14 hours ago [-]
> but LLMs have a place in vast number of places where we'd previously never use a classifier because it was too much effort.
Absolutely!
But that doesn't rule out that there are AIbros that will tell you to use an LLM, even if the task could be done with a static analyzer with a few rules. Use an LLM because it's AI!
bgroins 10 hours ago [-]
You seem to be quite fixated on a personality type rather than the underlying technology.
darkwater 8 hours ago [-]
Not really, although when a certain personality type dominates a scene, people tend to conflate the two. Especially when they are louder than the rest.
IanCal 15 hours ago [-]
There are absolutely good reasons to use smarter things for classification. Svm with tdf idf for example is super fast but will make very bad classifications at times in ways entirely unfixable with that system (thinking back to bats being an acronym for a theology degree classification or similar)
NitpickLawyer 14 hours ago [-]
This is a bit too radical and misses some nuance. In order to start building classifiers you at least need some ML domain knowledge, like what to google at least. Meanwhile, point anyone at the jev/open-jev-clones API and have them start classifying in <10 minutes. There's some value there. (same used to be valid for LLMs before jev became a thing. You can use llms as zero-shot classifiers with the right API wrapper, w/ no domain data).
msp26 10 hours ago [-]
Even if you have ml knowledge, it's still nice to be able to experiment with various use cases with zero upfront data/training time.
cryptonym 13 hours ago [-]
1000x more expensive run can be overall cheaper. You avoid upfront training/tuning cost. It all depends on your scale.
Kaedon 15 hours ago [-]
At ~$21,700 saved per million tickets, saving millions annually would suggest at least 92 million tickets a year to reach $2 million saved. Is that about right?
I initially thought of tickets as just customer support, or internal JIRAs, but the article mentions alarms, security findings, and capacity increase requests, so that makes the scale feel more plausible. Still surprising so many tickets go through the system.
dansalvato 8 hours ago [-]
In my past experience in IT, the ticketing system basically doubled as an aggregated event log, and it was a legitimately good use case. The monitoring software we ran on our thousands of client workstations and servers was extremely powerful and could capture any conceivable metric to use as a trigger for auto-generating a ticket.
So, a lot of tickets generated would simply just be data reports with some info dumped in the ticket description, designed to be auto-closed but possibly useful later as an easily-accessible timestamped reference to some data.
Lots of other auto-generated tickets were "probably not an issue but it should pass over the eyes of someone knowledgable about this server/network" tickets. Like Windows Event Log error roundups, CPU usage spikes, or specific triggers that were set up in the aftermath of some catastrophe to help preemptively avoid a repeat incident.
Anyway, tickets are a great "first line of filtration" where you can take a massive ingest of all different kinds of data (hardware metrics, system logs, network activity), apply meaningful human context ("This pattern could be a sign of xyz"), and store it in a unified database. With that use case, it's easy to imagine a company with millions of customers and millions of servers producing millions of tickets.
SoftTalker 7 hours ago [-]
I've never seen a ticket system used as just a dumping ground for data in auto-closed tickets. Seems like there would be better solutions for that, particulary for data you want to find later (in my experience, the search capability of ticket systems varies from poor to "don't bother, you won't find what you're looking for")
Anything else, that needs a human to interpret or take action on, yes, ticket systems get extremely wide usage.
dansalvato 2 hours ago [-]
Right, I didn't mean using tickets as a generic event log—the assumption is that someone would have set up triggers because they actually have a reason to have the data on hand, and it's a low-friction way of getting what they need without worrying about implementing new tools or systems. In a good human system, the trigger would be disabled once that info is no longer useful. (And in a bad human system it would just stick around until nobody knows what it's for anymore, haha.)
We also had tickets that would trigger scripts upon generation, e.g. an auto-generated "low disk space" ticket would then trigger a basic "disk cleanup" script, and close itself if enough space was freed to clear the threshold. Yes, it's comical that the ticket would do that—but hell, it worked great. It ran agnostically across thousands of differently-configured machines. And whether or not it was a human who clicked the "disk cleanup" button shouldn't determine the usefulness of having that event logged in the ticket database.
Point is, the whole system was a veritable Swiss Army knife that didn't necessarily have to equate a "ticket" (database entry) to "requires human intervention", provided there was a good human system in place to leverage the tool in a way that was productive for them.
julianlam 7 hours ago [-]
That makes sense, but a part of me is wary of introducing noise at such a scale and drowning out whatever signal remains.
Also ticketing systems in my (very) limited experience tend to be bloated pieces of software, so that adds a fun dimension to it.
dansalvato 3 hours ago [-]
Indeed, I learned pretty quickly that maintaining the line between "underreported information" and "information overload" is an indefinite challenge that evolves over time as context changes.
But the human team is what understands the big-picture context and the way it changes over time, so it makes sense for them to manage the triggers for the "this may be important" tickets—for them to then ingest and discern whether it's important. Ultimately, ticketing software is just a tool designed to distribute context-stamped information to humans who then discern what to do with it.
So I think just like all other productivity software, it's only as good as the human system in place that drives its usage. That explains the bloat as well—it's not its own system, it's a big old suite of tools that are each available to be implemented into a human system if desired, or ignored if not.
a012 10 hours ago [-]
When you’re enterprise size, every request needs a ticket otherwise it didn’t happen
mathgeek 10 hours ago [-]
Exactly. For folks working at smaller companies, think of it as "if you'd Slack someone to ask them to do something, that's a ticket" just purely because Slack doesn't scale to dozens of people needing to figure out which person even needs to get messaged in a given minute of the day.
red-iron-pine 31 minutes ago [-]
shared communications like slack also implies the same team, or at least the same company.
but i have 3 different support organizations, all of which are external. no ticket == no accountability, and that matters if and when things get ugly, like lawsuit territory level ugly
parthdesai 8 hours ago [-]
It's that, and you also want audit trail. But there's also processes for the sake of processes at Enterprise so a good % of these tickets aren't really needed but you need to file them to follow the process.
iLoveOncall 13 hours ago [-]
Yes, easily. I work for a "small" org in Amazon and it has more than 100,000 tickets a month. And it's a small fish in terms of number of tickets. As another commenter pointed out, tickets are used for basically everything administrative.
I checked just for fun and there are actually a LOT LOT more than 92M tickets in the past year.
swiftcoder 5 hours ago [-]
The amazon ticketing system used to run over email, and the exchange server team spent basically all their time trying to keep up with the scale...
f0cus10 15 hours ago [-]
yes, there's a ticket for EVERYTHING. people have tickets cut in their names even decades after they've left
fy20 14 hours ago [-]
At my company we also do the same every request to engineering (alarms, security, bug, customer support escalation) gets a ticket.
f0cus10 14 hours ago [-]
i mean there's even tickets cut if you need IT to fix a laptop or assign you a locker or unlock the code on your desk ped.
heck, there's even a ticket if the office toilet is clogged
jmbwell 7 hours ago [-]
It really helps. Tickets are cheap and persistent. Breezing by my office or shooting me a DM is cheap and ephemeral. I’m just going to respond with “do me a favor and put that in a ticket thanks” anyway
tippa123 14 hours ago [-]
I once had to raise a ticket for a double AA battery for my mouse. This sort of stuff frustrated me enough to start my own company.
phillc73 13 hours ago [-]
What does your company use for ticketing?
tippa123 10 hours ago [-]
Just spreadsheets for the time being. I think irrespective of the tool it's more the mindset for me. What is too small to have meticulously documented versus a general bucket. Also what becomes too big to not track on a ticket anymore but via project governance.
win311fwg 7 hours ago [-]
SqueakDesk
hyperbovine 11 hours ago [-]
Ah! You must be that energy harvesting mouse company.
baxtr 14 hours ago [-]
Were there tickets for tickets too?
tippa123 11 hours ago [-]
Yes, plus let's close this ticket and open a new one so that the KPIs look solid.
kakacik 12 hours ago [-]
If it works it ain't stupid. Big companies desperately need traceability and ownership. Way too many bullshitters playing politics, being loud on meetings etc but very little actual output.
oblio 13 hours ago [-]
In some big companied many core corporate teams which hate that level of transparency and build their own opaque systems. I've seen this for departments like IT, HR, legal, etc.
I'd rather have tickets for everything and full transparency except for confidential info.
iamflimflam1 12 hours ago [-]
Im a huge believer in transparency. But I also understand why, in a big company, people don’t like it. It’s very easy for metrics to be weaponised by political players who are out for themselves.
mwexler 8 hours ago [-]
Perhaps killing the need for tickets will be the real value unlock of llms. (Kidding! Kind of)
gofreddygo 3 hours ago [-]
essentially servicenow's business model.
responder41 3 hours ago [-]
based on my time at amazon it does seem low. Perhaps this is only for a subset, like a single VP-level org?
sjdh76 15 hours ago [-]
Well there is the UFO/Bigfoot/Poltergeist crowd.
If you dont fully pay attention to their theories about what they think is the reason for the suspicious stain on the ground in the parking lot, the queue is going to get flooded.
If your Boss belongs to that club oh man god say the division.
catchnear4321 11 hours ago [-]
it is a corporate mantra many of us want to forget.
it also functions as a response to any request.
“submit a ticket!”
vasco 15 hours ago [-]
Well if they use VPN point to point connections managed by AWS they'll get an email per week until they die for each VPN instance from AWS that they need to put in trash so I understand them.
zkmon 2 hours ago [-]
LLM is a hammer that makes you see everything as an LLM usecase.
Recently people started equating all cognitive abilities to an LLM. Every foundational work that happened in ML and data science prior to LLm, are largely forgotten in the flood of LLM hype.
A lot of usecases would be far better off using some NLP or decision systems, or rule engines rather than paying for Opus. Big companies blindly use expensive frontier moels as a blanket solution irrespective of usecase fit, because they are chasing the goal of being an "AI-powered" company. LLM killed all othe simpler, specific solutions.
ActorNightly 6 minutes ago [-]
From a technical perspective, all of the previous ML work that happened before LLMs is basically encapsulated in LLMs.
For exmaple, SVMs are exactly neural nets in representation. Transformers are Kernel Machines in esssence - take your sequence of tokens across n dimentions, and thats your higher level space where you can make decisions on the output. Gradient boosting/decision trees are your backpropagation and RL, except on a latent space rather than direct data output.
And you don't really need to use all the old techniques anymore given the local inference that exists. For example,
Both of those run quite fast on CPU, and can be used as a replacement for a lot of traditional techniques. For example, set up a system where user query first gets compared against instruction documents using embeddings, pick the best matching document, feed that instruction + user query into function gemma, which then generates a sequence of tool calls with the right inputs. Then finetune the instructions and/or function gemma for your use case.
The case of companies using Opus tokens for all things is a classic case of having enough profit to not have to worry about hyper optimization.
deepsquirrelnet 6 hours ago [-]
One of the biggest issues with using a general LLM like Sonnet is calibration. I think Jev really helped bring this to light. With Sonnet, you get an answer and that's it. It's really not that great for classification, especially tagging.
I've trained a lot of NLI models, which do similar things to decision models. A 1B parameter encoder is about the same capability as a 2B LLM, but if you have multiple tags to check, it's not as efficient because you can't cache the bidirectional attention. On the other hand, they do seem to do a bit better on calibration.
Decision models are much cheaper/faster to serve. For classification, you still need to do classic classification ML evaluation, but for many problem spaces you can get to ~90% of the power of a custom classifier model with an off the shelf decision model with just some prompt tuning.
deepsquirrelnet 4 hours ago [-]
Zero-shot models like Jev or others can still be difficult to prompt tune. A lot of times you are describing how you want things to be classified (called label verbalization), and miss nuances about how the model does interpretation of that.
I have started working on models that can both do a classification without explanation, but enable a "think" mode that briefly explains its decision. So if you want to keep track of failures and figure out why the model doesn't like them, then you have a direction to try fixing it.
I'd like to try wrapping this in some kind of GEPA-like process to automate tuning based on your labeled examples using the supervision of a larger reflection model.
This article is so confusing. For example, they say they tried the Strands Decider model and dismissed it for Gemma, then they continue to call the model they used Strands Decider? Did I miss something?
ninjalanternshk 12 hours ago [-]
This article explains it better (and rather colorfully)
They lopped the head off an existing Qwen or Gemma model and used its torso with their new head.
HarHarVeryFunny 8 hours ago [-]
As I understand it, Strands is basically an SDK (created by AWS then open sourced), so every solution is potentially custom.
Strands Decider is a decision model. maybe similar to Jev, that while based on an LLM is not one - it has a different "head" since it's not build to generate. The original prototype was based on (a modified) Qwen 3.5 2B, and the production version is based on Gemma 4 since that proved to be more accurate.
IMO the real news here is the big picture, not the details. Here's a high volume real world use case where a large expensive LLM (Claude Sonnet 4.6) has been replaced by a tiny 2B bespoke decision model.
Amazon is such a huge company that I'd guess they have internal business flow automation opportunities matching those of many other companies, so it's interesting to see this come out of AWS. Just as AWS itself was created by Amazon releasing something they has originally built for themselves, I wonder if we'll see more of these optimized real-world AI solutions coming out of Amazon/AWS/Strands?
ahoka 14 hours ago [-]
Not exactly sure, but as I understand this version of Strands is Gemma fine tuned for decision.
MisterMunchkin 11 hours ago [-]
It appears they just re-label other people’s models and call it a decision model.
Otterly99 8 hours ago [-]
It's baffling to me that a company like Amazon would choose to use a LLM for a task as simple as ticket classification. Don't they have thousands of engineers and petabytes of data? They even have their Mechanical Turk program if they want a cheap labelling workforce.
dansalvato 7 hours ago [-]
Ticket classification isn't necessarily so simple, and I can think of various reasons for using ML to classify tickets.
1. Some auto-generated ticket (e.g. a trigger from unusual hardware activity or a log entry) resembles a previous ticket that had meaningful action on it, addressed and solved by a specific team. That ticket could spend a week getting ignored or punted around when the right team can solve it in 5 minutes. ML could detect the similarity and reclassify it by that metric.
2. Human-submitted tickets are annoyingly fallible. For example, an "I can't log into my account" ticket could be auto-classified as "account recovery". Or maybe, e.g. the ticket system attaches a recent activity log from that IP address to be cross-checked by ML, which finds 5xx errors and reclassifies it for the technical team.
These are generic examples, but you can use your imagination and see how a company as huge as Amazon could benefit from this. I think it's for the purpose of minimizing the number of tickets that need to be manually reclassified because the wrong info was given to the wrong person.
infecto 8 hours ago [-]
I thought MT was being deprecated if not already?
It’s not that baffling Amazon has something like 70k+ engineers worldwide. Never worked there myself but in large orgs things sometimes get lost in the shuffle.
I don’t think timelines were shared but this reads like a pretty good case of building an MVP and then lowering the cost once it’s proven. One of the whole value props for LLMs from a ML perspective is being able to get something out the door a lot quicker than you would otherwise.
fusslo 4 hours ago [-]
I'd love some classification for support tickets where i work
whenever i submit a ticket (need hardware, problem with software, access, etc etc) I'm supposed to know what business unit im submitting to. There are 63 last time I checked. Some go to global IT. some go to local IT. some go to to procurement, etc etc. There's no resources on how to select the right one
I just guess and there is a dedicated person on every team who reassigns tickets to the right business unit.
nightmare.
an llm that parses my request and auto-populates in the back end would be cracker jack
iLoveOncall 25 minutes ago [-]
As someone working on tickets at Amazon, it's absolutely not straightforward at all and even an LLM (or a human!) will struggle.
QuinnyPig 8 hours ago [-]
They finally sunset Mechanical Turk.
TZubiri 1 hours ago [-]
>Mechanical Turk program if they want a cheap labelling workforce.
Do we have to point out why outsourcing a task that uses client data as input (the ticket) would be a bad idea?
psnehanshu 14 hours ago [-]
Ok, but why did they chose Sonnet in the first place? Couldn't they have chosen Haiku or GPT Luna or maybe even some small cheap Chinese model?
atestu 8 hours ago [-]
At the time of Sonnet 4.6 it was probably the best choice. Haiku was pretty dumb back then. Luna didn't exist.
Kyo91 8 hours ago [-]
Sonnet 4.6 was released 6 months before Luna (and Luna's major price cut came a month or so after that IIRC). So it's likely they started with Sonnet long before Luna was an option. I assume they evaluated Sonnet 4.6 vs Haiku at that time and found Haiku insufficient.
After that, I'm guessing it was just fairly low priority to update a working system. Amazon's net income last quarter was >$20B. Saving millions is huge on the scale of one or two engineers spending a week tackling it, but it's too small for the organization to make it a big initiative. By the time engineers decided to update it, decision models like Jev were popular and are naturally a better fit for the task than an LLM.
theshrike79 14 hours ago [-]
I’m guessing accuracy. Smaller models are just THAT much more inaccurate to not make it viable.
psnehanshu 12 hours ago [-]
I think for this purpose a small model will do reasonably well
theshrike79 6 hours ago [-]
But in some cases "reasonably well" is not good enough when you can get a "very good" or "excellent" by stepping up a model category.
I tried this with an AI subtitle translator thing I built for myself. Mid-tier models were considerably cheaper but "reasonably good" doesn't cut it in translations, the output language was too janky or used the wrong words. I had to bump it to Sol level (at the time) to get "good / very good" output, which was enough for something I intended to use once.
troyvit 7 hours ago [-]
What's weird is I didn't see mention of accuracy in the article.
wenc 7 hours ago [-]
Not really. Haiku is really not good at decision tasks (source: real experience).
rcstank 2 hours ago [-]
What about Haiku 5.5?
jeremyjh 9 hours ago [-]
Right. So we can either assume the engineers at Amazon are fucking clowns who don't do any engineering, or we can assume they tested and found that Sonnet was the cheapest model that met their reliability targets.
If you think the former is a likely hypothesis, then why did you even click on TFA?
vikramkr 8 hours ago [-]
Anthropic was the only frontier lab on bedrock for a while. Hate to break it to you but there are in fact many scenarios with reasons and pressures for non engineering-driven decisions to dominate
jeremyjh 4 hours ago [-]
But the question is also "Would Haiku not have worked?". If they ruled that much out, then they aren't going to be using anything super cheap. Also, they already had been hosting some of those cheap models on Bedrock and I'm sure they use them where it makes sense.
psnehanshu 9 hours ago [-]
Let's keep the discussion civil
ywvcbk 7 hours ago [-]
That's rather extreme? There is an possibility than Amazon's engineers are neither infallible nor idiots. Regardless of this specific situation highly talented engineers very often take decisions that lead to suboptimal outcomes on organizational level due to various reasons.
kevan 4 hours ago [-]
It's also possible this was optimal on the organizational level because of opportunity costs. You may be wasting $2M on inefficient compute but if your other opportunities are higher ROI and you're not allowed to add headcount then it sits. Disclaimer: At Amazon but no inside knowledge of this specific situation
catchnear4321 9 hours ago [-]
One can think the circumstances a circus without thinking the participants all to be clowns.
As for your question, seeing what others are doing helps one decide both what to do and what not to do.
jeremyjh 4 hours ago [-]
Good points, agreed.
driverdan 6 hours ago [-]
They should have used a classifier in the first place. Amazon has the data and resources to build a model or fine tune one. Using an off the shelf LLM doesn't make sense.
loafoe 13 hours ago [-]
Strands decider actually uses a small cheap/free Chinese model for its base architecture: Qwen
HarHarVeryFunny 8 hours ago [-]
Actually they started with Qwen 3.5, then switched to Gemma 4 since it was more accurate.
Hamuko 14 hours ago [-]
Not sure if it would even need to be a small Chinese model. Amazon would surely have the necessary hardware and infrastructure to run a bigger Chinese model with a low per-token cost too. They probably get their Sonnet at a discounted rate too since they do the inference "locally", but I imagine the savings for not having to pay Anthropic their share would still add up.
seizethecheese 3 hours ago [-]
> The world of agents gets a lot more exciting (and less costly) with decision models. If you’re running a classification agent, try the Strands Decider repo
Wait, their solution here is not an "agent" at all? It's literally output length of "1", unless I'm missing something? (It's not clear what they are replacing is an agent either, it looks like a reasoning model with no tools that outputs JSON.)
neronuser 2 hours ago [-]
I wonder when AWS adds decision models to Bedrock. It seems like a huge missed opportunity. In the blog they circumvent that via self-hosted decider model on the ec2
3eb7988a1663 15 hours ago [-]
Classifying into one of 20 groups? Did anyone try scikit learn?
ryankrage77 7 hours ago [-]
I did this at my last job. ComplementNB trained on ~30,000 examples (which were freeform text, and very unbalanced across categories) in under a minute on a laptop CPU, and with log-weighting of scores, was about 85% accurate. I don't remember how fast it was at evaluating the tickets beyond 'basically instant', as it was a proof-of-concept that got cancelled by changing ticketing system.
aitchnyu 11 hours ago [-]
Heard about classifiers a long time back. Will current ones resolve "reallyyyyyy beast mode" as a positive phrase like an LLM?
lofties 15 hours ago [-]
I was going to say, and I say this a little bit too often, a BERT model can do this even faster, with even less memory...
origini 13 hours ago [-]
So why did they not even try Haiku 5.5?
Why would Amazon go to another company to let them provide an adjusted decider model?
Amazon already has all these tickets labeld, they could just have build it themselves?
T4iga 12 hours ago [-]
They just didn't say. I think we can safely assume they are not completely stupid. They probably tested a few models a few months ago and have been using Sonnet since as it passed their benchmark/performed the best and then didn't change the system since, because it worked.
The blog post was focused on the switch to a decider model, not the initial architecture.
I think I'd be foolish to 'why didn't they just?', on the basis of <blank> because no mention was made in the post.
hobofan 13 hours ago [-]
What other company? Strands is Amazon (AWS).
origini 12 hours ago [-]
Oh i see.
Never heard of Strands before, felt complelty different when reading it more like a sales pitch from a consulting company
tdhz77 15 hours ago [-]
You can do this even more efficiently with the rust crate bonsai. You will have more control and you can create sceneries of logic. You can use incredibly cheap models this way and get way better results.
I built a real time character story. In which players interact and integrate fictional characters. There is no delay in response and the characters are all in character. Oh and it’s all voice.
It’s shocking how effective this is.
golf1052 14 hours ago [-]
I don't see how "bonsai + cheap model" is any better than the solution they used which was just "cheap open-source decision model that can be run internally at Amazon".
creativeSlumber 12 hours ago [-]
do you have a write up of this?
soon_to_be_sb 15 hours ago [-]
Where can I read more about this!?
choutianxius 3 hours ago [-]
Why not just tweak an open source model so that the final layer produces a probability vector and fine tune it?
oorestisime 9 hours ago [-]
Matches what I see in my own setup. Most agent calls don't need the frontier model, they need something fast that decides what happens next. Deepseek very underrated here as well. Then again, with the subscriptions the tokens still are pretty cheap so subagent delegation and improvements dont matter. Yet.
smcg 6 hours ago [-]
To generalize, I expect that specialized agents will start chipping away at the market share of the more general AI technologies. They are much cheaper and the market for them is maturing.
bilekas 10 hours ago [-]
I'm not very up to date on the Ai lingo, but is a Strands Decoder just a decision tree in the same vein as Jev ?
mdnahas 10 hours ago [-]
I am assuming that Strands Decoder is similar to Jev.
I do not believe Jev is a “decision tree”.
bilekas 10 hours ago [-]
Should have rephrased it as a "decision model" it's all a bit confusing terminology to keep up these days. "Noul" I guess!
gvv 14 hours ago [-]
What I don't understand is why are they not offering the decision model via Bedrock yet. Sagemaker / Lambda are quite expensive if you're not using the model continuously.
hubertzhang 14 hours ago [-]
People use machine learning models before and after LLM is invented.
egorfine 14 hours ago [-]
> JSON key order changed the answer. Sending the same content with keys in a different order moved the output probabilities by up to 0.19
Isn't it detrimental?
theahura 6 hours ago [-]
I read this as "saves millions of lives" and was very confused
slashdave 5 hours ago [-]
A few millions is nothing to Amazon's bottom line
fumeux_fume 8 hours ago [-]
They keep referring to the solution as an agent, but it's actually just a single call to a model.
iririririr 7 hours ago [-]
words mean nothing in the Slopcene
perarneng 4 hours ago [-]
Why is it not deployed on bedrock
docheinestages 10 hours ago [-]
No mention of quality, or at least I didn't see it. What seem to matter nowadays are speed and cost.
pluc 10 hours ago [-]
For 1 measly million I will say yes or no to anything.
andai 14 hours ago [-]
50x cheaper, damn. I recall Chinese LLMs being similarly cheaper (and quite suitable for such simple tasks).
I think a decision model is superior here based on the task, and of course they weren't out yet when they started, but I'm wondering if they wouldn't have similar results (on price, probably not on speed) if they went with DeepSeek or something?
I don't mean send the data to china obviously — there must be an American provider that offers it?
Anyway it's moot in this case (decision model is a better fit) but it just makes me wonder who else is overpaying by 2 orders of magnitude.
a1o 12 hours ago [-]
It does use qwen, just check its repository linked above
Kyo91 8 hours ago [-]
The blog specifies that they switched it to use Gemma for this task, but your point still stands.
llukas 14 hours ago [-]
Are developments like that priced in Anthropic IPO?
cmiles8 14 hours ago [-]
This pattern is playing out a lot. Running LLM inference at scale on apps typically lacks ROI. But I increasingly see folks use LLMs to help improve classical techniques.
So there’s some LLM use to develop a new version of algo, but maybe 2% of what would have been used by putting LLMs in the path of every API call. That generally does have ROI.
Putting LLMs in the loop each time is generally a dumb move, unless you’re an AI bro trying to get everyone to use LLMs for all the things.
nikit1411 14 hours ago [-]
yah i mean older models dosent mean they are worthless i still use cluade opus 4.6 for day to day task
iririririr 7 hours ago [-]
the biggest take sway here is that profit chances from the model provider to the generic computation provider.
the bubble is bubbling
Rapzid 11 hours ago [-]
Slop article. First paragraph is 100% AI! Fucking lazy ass slop drivel.
simianwords 13 hours ago [-]
why didn't they use Jev or Luna decision model? Seems odd that they used Sonnet and Strands instead of trying out of the box solutions
Kyo91 8 hours ago [-]
Strands Agents is an AWS product. They're dogfooding their own product in typical AWS fashion.
win311fwg 12 hours ago [-]
Because when Sonnet was originally chosen, Jev/Luna weren't available to choose from? The decision model movement hadn't happened yet. When they were ready to replace Sonnet, they already had their own decision model. Why use someone else's?
arbor-group 7 hours ago [-]
[flagged]
phillennium 15 hours ago [-]
It's funny how AI sometimes feels like it went full circle:
— "Let's build efficient classifiers"
— "Forget about the silly classifiers, LET'S BUILD A MACHINE GOD"
— "Ah, that machine god is often too expensive and underwhelming. Let's build efficient classifiers"
Schiendelman 15 hours ago [-]
The key is that the circle takes a few weeks instead of a decade.
written-beyond 14 hours ago [-]
You'll be surprised how fast things move when there's European GDP levels of investment going into an industry.
oblio 13 hours ago [-]
You should probably say "European country GDP levels of investment" as the European economy as whole is +/- the size of the US economy.
solidasparagus 14 hours ago [-]
If you were actually involved in that work pre-LLM, you know it didn't go in a circle. Most projects died at training data and problem definition. That isn't the case with LLM applications which can ship with zero data collection and labelling effort.
Now you can iterate on your 20 ticket categories in days against real workloads and not commit to training an efficient model until you've proven the prototype in production.
dejj 11 hours ago [-]
"Let them eat cake"-and-find-out-phase is what I'd like to call step 2.
tveita 13 hours ago [-]
> One thing that surprised us was that JSON key order changed the answer. Sending the same content with keys in a different order moved the output probabilities by up to 0.19.
Or is this the Claudish "One thing that suprised us" that just means "and here's another thing"?
creativeSlumber 12 hours ago [-]
I did not know that json order can change the answer. Doesn't that make LLM's unreliable? Do we know why this happens?
xolox 11 hours ago [-]
I think this is similar to how small changes in phrasing can completely change the answer you get from an LLM. This all goes to confirm that LLMs are, at their core, still text completion engines, they (frequently / always?) don't have a firm semantic grip on the topic at hand.
Here's an interesting reference on this that gives a bit more background information:
-- 8< --
Because LLMs (and many other ML systems) are chaotic, it is possible to manipulate them into doing something unexpected through a small, apparently innocuous change to their input. These changes can be illegible to human observers, which makes them harder to detect and prevent.
For example, flipping a single pixel in an image can make computer vision systems misclassify images. You can replace words with synonyms to make LLMs give the wrong answer, or introduce misspellings or homoglyphs. You can provide strings that are tokenized differently, causing the LLM to do something malicious. You can publish poisoned web pages and wait for an LLM maker to use them for training. Or sneak invisible Unicode characters into open-source repositories or social media profiles.
If these were tickets generated internally (like monitors, alarms), why not deterministically tag them at the source and only invoke the LLM when it's ambiguous? The Amazon I knew had an appetite for efficiency that seems to have been blinded by AI euphoria.
mannyv 4 hours ago [-]
What's interesting to me is that they were able to deploy something so fast. Sonnet 4.6 was released in February 2026, went into production, then was replaced by "The Decider."
I guess some parts of amazon still can move fast.
speedboat199 14 hours ago [-]
Meanwhile they could have just switched to Haiku 5.5 to get same savings. No need for complex orchestration involving GPU instances and would have just been a one line change in the code.
HarHarVeryFunny 8 hours ago [-]
The Strands solution is based on an open weights model, so has zero API cost, and Amazon are obviously well positioned to serve it themselves extremely inexpensively.
A modern model like Haiku 5.5 is still going to be overkill for many of these decision model use cases, and is probably at least 10x the number of parameters (Haiku 20-50B ??? vs Strand Decsions 2B Gemma 4).
ZiiS 13 hours ago [-]
Amazon saves millions by asking their pencil manufacture to make them 1mm shorter. At that scale anything adds up to millions.
psadri 5 hours ago [-]
I wonder how much they would have saved if they simply used gpt-6-luna instead. luna is 30x cheaper. and now you don't have tech debt in the form of a custom trained model that needs to be maintained etc…
pietz 9 hours ago [-]
Wait, are you telling me that if you replace an oversized and outdated model, with a small one that fits the use case and reduce your input by 70%, you end up saving money?!??
polynomial 9 hours ago [-]
Yes, you save $20,000.
GuB-42 5 hours ago [-]
Amazon saves millions of dollars by replacing Sonnet 4.6 with Strands Decider
I know some people take the existential risk of AI seriously, but Sonnet 4.6 is not going to kill millions of people, right?
coolgoose 16 hours ago [-]
Yea right... Paying market price of 3 usd per mil tokens on sonnet 4.6...
0x00cl 14 hours ago [-]
Sounds like a bunch of `if` statements with extra steps. Maybe throw an RNG decider too.
milimetric 10 hours ago [-]
ooh, this is gonna hurt later:
"except that on long tickets Strands Decider drops the oldest comments to fit."
milimetric 10 hours ago [-]
ooh, this'll hurt later:
"except that on long tickets Strands Decider drops the oldest comments to fit."
lencastre 15 hours ago [-]
counting the miliseconds until the insurance industry picks this up
Oras 14 hours ago [-]
Nice marketing to sell their decision model.
12 hours ago [-]
tiffanyh 3 hours ago [-]
“Company selling product claims their customers save money.”
News at 11.
OtomotO 12 hours ago [-]
If you use the right tool for the job at hand, you save money.
Of course you can try to dig a hole with a hammer instead of a shovel...
nickphx 5 hours ago [-]
whats old is new again, so amazing and innovative... what will they think of next??
oofbey 16 hours ago [-]
If their model costs 98% less than Sonnet, and Amazon is saving “millions” then these folks are raking in 2% of millions. Congrats I guess?
mcintyre1994 15 hours ago [-]
I think you’re missing that Strands Agents are open source models released by AWS, not an external startup that Amazon is paying.
otterley 14 hours ago [-]
You're (understandably) a bit mixed up here. Strands Agents is an agent SDK. Strands Decider is a decision model.
swiftcoder 15 hours ago [-]
Yes, this is more of less an advert, using their internal adoption as a case study
sant-sh 12 hours ago [-]
that's what it looks like. they are trying to showcase the usecase here
rrr_oh_man 15 hours ago [-]
I yearn for the day of going back to make v. buy, not buy v. buy
15 hours ago [-]
vincnetas 16 hours ago [-]
2% of millions ... so far. Take in to account growth potential.
Den_VR 16 hours ago [-]
small, open source, decision model optimized for fast experimentation, local development, and innovation...
But just think of the street cred!
sarathkavuru 8 hours ago [-]
[flagged]
raz32dust 5 hours ago [-]
[dead]
ankur_misra 7 hours ago [-]
[flagged]
unnataisolution 13 hours ago [-]
[flagged]
GordyNumberOne 14 hours ago [-]
[dead]
14 hours ago [-]
novalyn_ai 9 hours ago [-]
[flagged]
geberele 6 hours ago [-]
[flagged]
thenayr 6 hours ago [-]
[dead]
newsicanuse 2 hours ago [-]
Almost anything hyped by or lead by Swaminathan Sivasubramanian at Amazon is as close to scam or bar lowering product as it can be.
nlpnerd 15 hours ago [-]
Founder of Seldon AI (www.seldon-ai.com) here. We are working on dev tools to help developers with this kind of refactoring and continual production work.
Basically there is a whole host of tarpit usage of LLMs in application backends today. This is simply another case of LLMs enabling people to build what they cannot productionize or maintain.
Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.
The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.
They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".
These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.
I think system one models is a sensible paradigm, and we are on the precipice of something great, but there is a lot more work needed above the model layer to make this work.
Rapzid 10 hours ago [-]
@dang why can't I down vote this?
Asking for a friend.
No, serious question; there are no downvote controls on this "advertisement", what gives?
DoctorOW 35 minutes ago [-]
I had no problem downvoting it
10 hours ago [-]
sznio 16 hours ago [-]
why didn't they upgrade to further sonnet versions?
Zanfa 15 hours ago [-]
If they’re running on AWS Bedrock, lack of batch API support for more recent Anthropic models would be my guess.
devld 7 hours ago [-]
4.6 was peak Sonnet.
mikae1 15 hours ago [-]
How about at least reading a couple of paragraphs from the post this thread is about before commenting? :)
nalekberov 13 hours ago [-]
As a customer I don’t care what model they are using. Amazon is one of the worst e-commerce platforms (at least here in Germany) - damaged boxes, damaged items, items taking forever to ship, having to deal with rude delivery people just to name a few.
This is perfect example of how to not approach solving business problems. What do they actually need? Listening to customers.
What are they doing? Outsourcing delivery to third parties, lowering quality controls, creating worst working conditions, filling the platform with sponsored listings and ads, allowing any sort of cheap Chinese garbage with fake 5 stars on the platform, making the UI worse and worse everyday, and the list goes on and on…
iLoveOncall 12 hours ago [-]
> Amazon is one of the worst e-commerce platforms (at least here in Germany) - damaged boxes, damaged items, items taking forever to ship, having to deal with rude delivery people just to name a few.
I lived in Luxembourg for 4 years, which you have to order through amazon.de and is fulfilled by amazon.de, and I've never had any of the issues you describe. And that's with an average of 120 orders per year.
nalekberov 12 hours ago [-]
Just because you aren't running into problems doesn’t mean the platform is perfect; it might simply mean your standards are lower.
Use some logic before attacking: Luxembourg is only the size of Saarland (German state). Fewer orders and fewer deliveries naturally mean fewer problems.
iLoveOncall 12 hours ago [-]
> Fewer orders and fewer deliveries naturally mean fewer problems.
How about you read before attacking? I said it's pooled with all german orders.
Ylpertnodi 9 hours ago [-]
A bit further south, here...never had a problem apart from reconciling the moral side of things. But, on Tuesday i had to order off am as no local store would price match. Item arrived friday. I was in one of the stores just a few mins ago...they now have the amazon price showing.
I let stores know they mess up, so an email will be sent to $tore.
As for rude delivery - have stuff dellivered to a drop-off point. Mine is my local newsagent. Very slim chance of them ever being rude.
Rendered at 22:13:04 GMT+0000 (Coordinated Universal Time) with Vercel.
[1] "Linnaeus: A highly reusable and adaptable ML based log classification pipeline", https://arxiv.org/abs/2103.06927
NOTE: the work was actually accepted at the International Conference on Software Engineering (ICSE) in 2021.
An issue in industry was that companies would not allow training on their data across customers. Understandably so, as SVM dimensions are readily interpretable given some thought. General purpose datasets were ill-suited too depending on the task.
So you'd start from scratch every time, annotate features like crazy, and often didn't have enough training data overall.
LLMs "solved" that by just ignoring the previous rules and training on everything, which lets the necessary world knowledge seep back in.
I'm sure we'll see many combinations of LLMs and regular machine intelligence in the near future.
If you want to make a ticket classification model, you don't need to train across many customers doing ticket classification, who might not want you to train on their data and whose data is impossible to obtain unless you already are a large company. Instead, you can train on a completely different task which forces the model to learn the syntactic and semantic structure of the English language — say predicting missing words in Wikipedia articles, which you can't do successfully without a good understanding of English — and then fine-tune that model on your specific task with much less data.
Because the initial task is so general, you can have one company do the expensive pretraining, share the base model, and then have everybody else just do the finetuning on their (small) proprietary dataset.
LLMs and models like Jev go one step further, letting you use them as classifiers with no training at all, with nothing but rules in the prompt. And with a few tricks, you can indeed use them to get pretty high-quality labels, which you can then distill into a fine-tuning dataset for a much smaller model.
Word2vec is a two layer perceptron.
I feel like people in the field forgot everything.
True, but could be taken the wrong way. LLMs were also not able to train on that private customer data. They trained on so much general public data that they became capable enough to operate generally even on private data they had not been trained on.
That was the big breakthrough. I know that’s what you were trying to say, but on sites like this there are a lot of people who will see a comment like that and assume that LLMs like Sonnet were fed all of the private customer data that was withheld from training other models.
[1] I went down the rabbit hole of using sequence kernels quite a bit back then, see https://arxiv.org/abs/1910.13800, but the timing was terrible. RNNs and variants like LSTMs were beginning to become popular, and everyone (including me) switched.
[2] https://pocket-text-lab.abhishek-ghose-82.workers.dev/
You can also do stuff like setup active learning but have the labeller be an llm as well to make things more efficient.
Definitely worth it for PoC level things if not full ones.
Edit - also reviewing, so it can run training, test, review failure cases and investigate itself. You can leave them to plough through a few iterations where the early part is the classic time you’d have a few obviously bad sets of results.
Building an SVM isn't a tricky job, but you probably need a data scientist involved, and to think carefully about a labeled dataset.
I think we're seeing the modelling equivalent of something like electron apps. Where they waste a tonne of resources, but make an easier developement experience.
Will be interesting to see how much or little this changes with LLM costs.
So between llm and pseudo experts, at this stage, I prefer to go with a LLM.
Well not anymore... You can just get AI to build it.
Just like, LLMs can build web back ends, but I'd want a security professional to give it some kind of thumbs up. Especially if a lot of money rides on it.
Also IMO it's a lot easier to skim something and say "yeah that looks like it does what we want" than it is to come up with it from scratch.
Your model only took minutes to train if you don't count the data collection and labelling work
Saying that the field got reduced to only LLMs is like saying that the field of combustion engines got reduced to cars. The trucks, combine harvesters and peaker plants are still here, they're just in the background, invisible to people who are not part of the industry.
Banks still use random forests and SVMs, medicine and image processing still use CNNs (and people who take photos with their phone don't even realize how much image processing is being done), TTS, ASR and predictive typing models are in wide use, and there's probably still plenty of BERT in spam classification and such.
Yeah, exactly. Your phone just...does stuff. Bar/QR code scanning is probably using a CNN to minimize resource utilization.
LLMs are like an ice breaker to talk about AI and all the ways it has existed for decades, and what AI really refers to.
In software engineering, we learned a long time ago that giant monoliths are not the answer. You want smaller, but specialized units that work together. They perform better at their specific task, are more maintainable, you can just upgrade one part of the system without having to impact the others, ....
I don't know where the focus on building a single model that can do everything comes from.
That depends on the question. If the question is "How do I utilize 10,000 software engineers?" then yes, a monolith is not the answer. However, this isn't a problem most companies face. Starting out with microservices is rarely a good idea either as you'll have a hard time getting the boundaries right.
So if that’s a part of the future, what’s the purpose of spending billions of dollars on large AI models? Is this just a continuation of the story that there is no moat? For the big model makers?
Why do you need a master of everything? when you can just use a much smaller specialist AI model.
If you want a model that can do X but don't care about Y And Z, intuition says you only have to train the model on data relating to X. However, early LLM research showed that doing this results in a far less X-capable model than one trained on data for Y and Z as well.
I would assume a smaller model trained specifically to produce code based on natural language input would perform at least just as well, which would already be better given the smaller size / probably lower training resource requirements. This is of course, as already said, an assumption. But generalizing typically comes at a cost, there is no reason why this wouldn't apply to LLMs, it even applies to humans.
That might mean that spending all that money on that giant master of none model might lose out to the much smaller specialist model, uh-oh…
The Bitter Lesson is that human-written rule models will always lose out to trained models in the end.
Coding an exercise app will benefit from a model that can code and understands what exercise is. A car showroom website will benefit from a model that understands what relevant filters would be.
Nah, let’s stand firm here and not give in so easily. Parent suggests a coding model needs exercise info in case you want to code an exercise model? Absurd, that leads to the know everything monolith to do anything.
Language, reasoning, tool use, and domain knowledge.. in this case the domain is coding. Someone wants an exercise app? Great, so we look that shit up, we don’t bake it in. We don’t want to bake anything we don’t need in.
We don’t even need to explain this completely reasonable and totally normal engineering preference that small is beautiful. it’s the other side that needs to justify wtf exercise has to do with coding or why everything from twitter posts to a working knowledge of 15th century French poetry just might be helpful for disconnected domains, like say serious work in contemporary astrophysics. Back to basics, No one asked for that, No one should accept as given that this is the way anything needs to work
No, the point is having broader knowledge than just "can code" is important for building software for humans to use.
> We don’t even need to explain this completely reasonable and totally normal engineering preference that small is beautiful.
It is, of course it is, as long as that small thing is useful. My argument is not "let's put literally everything in this thing just in case" it's "broader world knowledge is important for building many things".
Also, we don't look at CVs or histories of people and say "hmm, this person could be good at coding for us but I really wish they didn't have such a broad and expansive history".
> Great, so we look that shit up
But what do you have it look up? What is exercise? What is a muscle and what is a human? How is it going to understand those references? Does it have the broad enough understanding to see where the gaps are?
At some level unless all you want is "loop over the array and sum the results into a variable called sales_total" converted into code, you have to have some broader knowledge about things.
And once you start considering that maybe it needs more than simply code, we're not disagreeing we're just discussing where the line should be. Which I've also not placed, because there's no simple statement to make on that.
Damn right we don't say that. And I don't look at a carpenter and wonder if his head is hard enough to drive nails like a hammer either. People aren't tools, and tools aren't people. You've conflated them because they are both capable of doing work. And that's exactly the type of confusion and category error that the perspective I'm arguing against will lead to.
Wanting a multipurpose tool or system is occasionally reasonable. Much more often it is very, very to close misunderstanding the nature of a problem, what a tool is, or what a system should be. The particulars of purpose and mechanism of action matter, which is why a blowtorch is sometimes a wrench but only if you're kind of desperate. It is always a very scandalous thing to speak ill of someone's best new electric banana-phone / dishwasher-piano that they want to sell you. But that just sounds messy to me, I'm not sure it will improve the sound quality or taste of anything involved.
Does it? All of the latest frontier models that have made technical details publicly available are using MoE; smaller specialized units that work together.
LLMs are the powerful general-purpose CPU of the ML world. Using an LLM as a classifier is woefully inefficient compared to using a dedicated classifier. On the other hand, that LLM exists right now and you can hook it up in ten seconds. You could ask: why use an LLM when a dedicated classifier would be so much more efficient? You could also ask: why build a dedicated classifier when a sufficiently capable LLM is available off the shelf? Neither one is the right answer every time.
"AGI"
Does that include collecting, curating and preparing the training dataset? Because that's kinda the hard part I find.
When I asked why we don't build some NLP tagging which would be cheaper and run in a fraction of time, the lead told me, "What if we need to use more unstructured data? GPT can handle that without any extra work."
The team has yet to make any other changes to the pipeline, but in the two-ish years since then, they've spent a ton of money and wasted countless hours on delayed processing because of it. Some day, maybe, they'll learn that traditional ML techniques are superior in virtually every way to just throwing data at an LLM to see what sticks.
With the LLM you don't even need a data scientist to train the model, you just need someone who can connect to APIs. And the LLM is fine with adding and removing categories. No retraining.
It makes the initial faces incredibly fast. Then it is slow and expensive to run, so once the system settles, it becomes the target for model optimization, e.g, a dedicated, or semi-dedicated model.
You empathize the set up cost but ignore that the model is in the magnitude of a million x in size and compute cost of a classifier model. That's like comparing a CPU with an IC
I see coworkers using AI for tasks like "visit an API endpoint, download a file, put it in S3". This is so stupid. Okay if you want to use AI, but use it to write python code to do that, why pay in tokens every time you want to run that task, it makes no sense
Your S3 case tho I think is a bit misplaced. You're right that if all your keys are configured correctly (API key, S3 key, etc) that a script to do that is like 5 lines in Python (once you get your venv going or use uv to setup the project lol), but tell an agent to just do the thing, and it will make sure that your API keys are in place and that you have access to the S3 bucket you're trying to write to and that the whole thing is in a venv, and then just report back that it's done.
Is it wasteful? Sure. For years I used to feel that way when I saw someone using a messaging app to message a friend across the hall in a loud room or when I saw someone taking a phone picture of their computer monitor and sending it. In the former case your packets are literally going from your phone to a central server and then onto someone else's phone in the same room. But, well, convenience wins.
Probably under the hood Claude creates some temporary python code or something.
But it bothers me, I don't know why. It's like paying a subscription service for the most stupid tasks
That is the reason.
Once you then have a system that is used enough, it starts making sense to look to replace that with a leaner classifier, but LLMs have a place in vast number of places where we'd previously never use a classifier because it was too much effort.
Absolutely! But that doesn't rule out that there are AIbros that will tell you to use an LLM, even if the task could be done with a static analyzer with a few rules. Use an LLM because it's AI!
I initially thought of tickets as just customer support, or internal JIRAs, but the article mentions alarms, security findings, and capacity increase requests, so that makes the scale feel more plausible. Still surprising so many tickets go through the system.
So, a lot of tickets generated would simply just be data reports with some info dumped in the ticket description, designed to be auto-closed but possibly useful later as an easily-accessible timestamped reference to some data.
Lots of other auto-generated tickets were "probably not an issue but it should pass over the eyes of someone knowledgable about this server/network" tickets. Like Windows Event Log error roundups, CPU usage spikes, or specific triggers that were set up in the aftermath of some catastrophe to help preemptively avoid a repeat incident.
Anyway, tickets are a great "first line of filtration" where you can take a massive ingest of all different kinds of data (hardware metrics, system logs, network activity), apply meaningful human context ("This pattern could be a sign of xyz"), and store it in a unified database. With that use case, it's easy to imagine a company with millions of customers and millions of servers producing millions of tickets.
Anything else, that needs a human to interpret or take action on, yes, ticket systems get extremely wide usage.
We also had tickets that would trigger scripts upon generation, e.g. an auto-generated "low disk space" ticket would then trigger a basic "disk cleanup" script, and close itself if enough space was freed to clear the threshold. Yes, it's comical that the ticket would do that—but hell, it worked great. It ran agnostically across thousands of differently-configured machines. And whether or not it was a human who clicked the "disk cleanup" button shouldn't determine the usefulness of having that event logged in the ticket database.
Point is, the whole system was a veritable Swiss Army knife that didn't necessarily have to equate a "ticket" (database entry) to "requires human intervention", provided there was a good human system in place to leverage the tool in a way that was productive for them.
Also ticketing systems in my (very) limited experience tend to be bloated pieces of software, so that adds a fun dimension to it.
But the human team is what understands the big-picture context and the way it changes over time, so it makes sense for them to manage the triggers for the "this may be important" tickets—for them to then ingest and discern whether it's important. Ultimately, ticketing software is just a tool designed to distribute context-stamped information to humans who then discern what to do with it.
So I think just like all other productivity software, it's only as good as the human system in place that drives its usage. That explains the bloat as well—it's not its own system, it's a big old suite of tools that are each available to be implemented into a human system if desired, or ignored if not.
but i have 3 different support organizations, all of which are external. no ticket == no accountability, and that matters if and when things get ugly, like lawsuit territory level ugly
I checked just for fun and there are actually a LOT LOT more than 92M tickets in the past year.
heck, there's even a ticket if the office toilet is clogged
I'd rather have tickets for everything and full transparency except for confidential info.
If you dont fully pay attention to their theories about what they think is the reason for the suspicious stain on the ground in the parking lot, the queue is going to get flooded. If your Boss belongs to that club oh man god say the division.
“submit a ticket!”
Recently people started equating all cognitive abilities to an LLM. Every foundational work that happened in ML and data science prior to LLm, are largely forgotten in the flood of LLM hype.
A lot of usecases would be far better off using some NLP or decision systems, or rule engines rather than paying for Opus. Big companies blindly use expensive frontier moels as a blanket solution irrespective of usecase fit, because they are chasing the goal of being an "AI-powered" company. LLM killed all othe simpler, specific solutions.
For exmaple, SVMs are exactly neural nets in representation. Transformers are Kernel Machines in esssence - take your sequence of tokens across n dimentions, and thats your higher level space where you can make decisions on the output. Gradient boosting/decision trees are your backpropagation and RL, except on a latent space rather than direct data output.
And you don't really need to use all the old techniques anymore given the local inference that exists. For example,
https://ai.google.dev/gemma/docs/functiongemma
https://blog.google/innovation-and-ai/technology/developers-...
Both of those run quite fast on CPU, and can be used as a replacement for a lot of traditional techniques. For example, set up a system where user query first gets compared against instruction documents using embeddings, pick the best matching document, feed that instruction + user query into function gemma, which then generates a sequence of tool calls with the right inputs. Then finetune the instructions and/or function gemma for your use case.
The case of companies using Opus tokens for all things is a classic case of having enough profit to not have to worry about hyper optimization.
I've trained a lot of NLI models, which do similar things to decision models. A 1B parameter encoder is about the same capability as a 2B LLM, but if you have multiple tags to check, it's not as efficient because you can't cache the bidirectional attention. On the other hand, they do seem to do a bit better on calibration.
I keep a leaderboard here: https://huggingface.co/spaces/dleemiller/open-nli-leaderboar...
I have started working on models that can both do a classification without explanation, but enable a "think" mode that briefly explains its decision. So if you want to keep track of failures and figure out why the model doesn't like them, then you have a direction to try fixing it.
I'd like to try wrapping this in some kind of GEPA-like process to automate tuning based on your labeled examples using the supervision of a larger reflection model.
https://huggingface.co/dleemiller/AbsoluteZero-nli-4b
https://strandsagents.com/blog/introducing-strands-decider/
They lopped the head off an existing Qwen or Gemma model and used its torso with their new head.
Strands Decider is a decision model. maybe similar to Jev, that while based on an LLM is not one - it has a different "head" since it's not build to generate. The original prototype was based on (a modified) Qwen 3.5 2B, and the production version is based on Gemma 4 since that proved to be more accurate.
IMO the real news here is the big picture, not the details. Here's a high volume real world use case where a large expensive LLM (Claude Sonnet 4.6) has been replaced by a tiny 2B bespoke decision model.
Amazon is such a huge company that I'd guess they have internal business flow automation opportunities matching those of many other companies, so it's interesting to see this come out of AWS. Just as AWS itself was created by Amazon releasing something they has originally built for themselves, I wonder if we'll see more of these optimized real-world AI solutions coming out of Amazon/AWS/Strands?
1. Some auto-generated ticket (e.g. a trigger from unusual hardware activity or a log entry) resembles a previous ticket that had meaningful action on it, addressed and solved by a specific team. That ticket could spend a week getting ignored or punted around when the right team can solve it in 5 minutes. ML could detect the similarity and reclassify it by that metric. 2. Human-submitted tickets are annoyingly fallible. For example, an "I can't log into my account" ticket could be auto-classified as "account recovery". Or maybe, e.g. the ticket system attaches a recent activity log from that IP address to be cross-checked by ML, which finds 5xx errors and reclassifies it for the technical team.
These are generic examples, but you can use your imagination and see how a company as huge as Amazon could benefit from this. I think it's for the purpose of minimizing the number of tickets that need to be manually reclassified because the wrong info was given to the wrong person.
It’s not that baffling Amazon has something like 70k+ engineers worldwide. Never worked there myself but in large orgs things sometimes get lost in the shuffle.
I don’t think timelines were shared but this reads like a pretty good case of building an MVP and then lowering the cost once it’s proven. One of the whole value props for LLMs from a ML perspective is being able to get something out the door a lot quicker than you would otherwise.
whenever i submit a ticket (need hardware, problem with software, access, etc etc) I'm supposed to know what business unit im submitting to. There are 63 last time I checked. Some go to global IT. some go to local IT. some go to to procurement, etc etc. There's no resources on how to select the right one
I just guess and there is a dedicated person on every team who reassigns tickets to the right business unit.
nightmare.
an llm that parses my request and auto-populates in the back end would be cracker jack
Do we have to point out why outsourcing a task that uses client data as input (the ticket) would be a bad idea?
After that, I'm guessing it was just fairly low priority to update a working system. Amazon's net income last quarter was >$20B. Saving millions is huge on the scale of one or two engineers spending a week tackling it, but it's too small for the organization to make it a big initiative. By the time engineers decided to update it, decision models like Jev were popular and are naturally a better fit for the task than an LLM.
I tried this with an AI subtitle translator thing I built for myself. Mid-tier models were considerably cheaper but "reasonably good" doesn't cut it in translations, the output language was too janky or used the wrong words. I had to bump it to Sol level (at the time) to get "good / very good" output, which was enough for something I intended to use once.
If you think the former is a likely hypothesis, then why did you even click on TFA?
As for your question, seeing what others are doing helps one decide both what to do and what not to do.
Wait, their solution here is not an "agent" at all? It's literally output length of "1", unless I'm missing something? (It's not clear what they are replacing is an agent either, it looks like a reasoning model with no tools that outputs JSON.)
Why would Amazon go to another company to let them provide an adjusted decider model?
Amazon already has all these tickets labeld, they could just have build it themselves?
The blog post was focused on the switch to a decider model, not the initial architecture.
I think I'd be foolish to 'why didn't they just?', on the basis of <blank> because no mention was made in the post.
Never heard of Strands before, felt complelty different when reading it more like a sales pitch from a consulting company
I built a real time character story. In which players interact and integrate fictional characters. There is no delay in response and the characters are all in character. Oh and it’s all voice.
It’s shocking how effective this is.
I do not believe Jev is a “decision tree”.
Isn't it detrimental?
I think a decision model is superior here based on the task, and of course they weren't out yet when they started, but I'm wondering if they wouldn't have similar results (on price, probably not on speed) if they went with DeepSeek or something?
I don't mean send the data to china obviously — there must be an American provider that offers it?
Anyway it's moot in this case (decision model is a better fit) but it just makes me wonder who else is overpaying by 2 orders of magnitude.
So there’s some LLM use to develop a new version of algo, but maybe 2% of what would have been used by putting LLMs in the path of every API call. That generally does have ROI.
Putting LLMs in the loop each time is generally a dumb move, unless you’re an AI bro trying to get everyone to use LLMs for all the things.
the bubble is bubbling
— "Let's build efficient classifiers"
— "Forget about the silly classifiers, LET'S BUILD A MACHINE GOD"
— "Ah, that machine god is often too expensive and underwhelming. Let's build efficient classifiers"
Now you can iterate on your 20 ticket categories in days against real workloads and not commit to training an efficient model until you've proven the prototype in production.
That's like... really basic LLM knowledge? Like how e.g. this article measures how often answers change based on order https://zachlandes.com/trolley-problem/#appendix-a:~:text=Ch...
Or is this the Claudish "One thing that suprised us" that just means "and here's another thing"?
Here's an interesting reference on this that gives a bit more background information:
-- 8< --
Because LLMs (and many other ML systems) are chaotic, it is possible to manipulate them into doing something unexpected through a small, apparently innocuous change to their input. These changes can be illegible to human observers, which makes them harder to detect and prevent.
For example, flipping a single pixel in an image can make computer vision systems misclassify images. You can replace words with synonyms to make LLMs give the wrong answer, or introduce misspellings or homoglyphs. You can provide strings that are tokenized differently, causing the LLM to do something malicious. You can publish poisoned web pages and wait for an LLM maker to use them for training. Or sneak invisible Unicode characters into open-source repositories or social media profiles.
-- >8 --
Source: https://aphyr.com/posts/412-the-future-of-everything-is-lies...
I guess some parts of amazon still can move fast.
A modern model like Haiku 5.5 is still going to be overkill for many of these decision model use cases, and is probably at least 10x the number of parameters (Haiku 20-50B ??? vs Strand Decsions 2B Gemma 4).
I know some people take the existential risk of AI seriously, but Sonnet 4.6 is not going to kill millions of people, right?
"except that on long tickets Strands Decider drops the oldest comments to fit."
"except that on long tickets Strands Decider drops the oldest comments to fit."
News at 11.
Of course you can try to dig a hole with a hammer instead of a shovel...
But just think of the street cred!
Basically there is a whole host of tarpit usage of LLMs in application backends today. This is simply another case of LLMs enabling people to build what they cannot productionize or maintain.
Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.
The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.
They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".
These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.
I think system one models is a sensible paradigm, and we are on the precipice of something great, but there is a lot more work needed above the model layer to make this work.
Asking for a friend.
No, serious question; there are no downvote controls on this "advertisement", what gives?
This is perfect example of how to not approach solving business problems. What do they actually need? Listening to customers.
What are they doing? Outsourcing delivery to third parties, lowering quality controls, creating worst working conditions, filling the platform with sponsored listings and ads, allowing any sort of cheap Chinese garbage with fake 5 stars on the platform, making the UI worse and worse everyday, and the list goes on and on…
I lived in Luxembourg for 4 years, which you have to order through amazon.de and is fulfilled by amazon.de, and I've never had any of the issues you describe. And that's with an average of 120 orders per year.
Use some logic before attacking: Luxembourg is only the size of Saarland (German state). Fewer orders and fewer deliveries naturally mean fewer problems.
How about you read before attacking? I said it's pooled with all german orders.