Decider v1.3 and Decider v.1.2 seem to have same benchmarks. However, V1.3 flash beats V1.2 flash by 4 points.
I am not particulary excited about this pattern/trend of having flash versions for decision models. The whole point of Decision models was to deliver faster ouput for high volume requests at much cheaper cost than Reasoning models. Since most reasoning models can and probably do outperform decision models already in accuracy but they are a lot slower and are pricier. Reasoning Models can already return structured ouput.
So your flagship model must be fastest and most accurate.
Unless it can handle some different form of input other than just image and text, I don't see the point of having both flagship and flash version for the same model.
deepsquirrelnet 1 days ago [-]
> So your flagship model must be fastest and most accurate.
That's rarely the way anything in life works though. Do you prioritize fastest OR most accurate? That's a question that depends on context, task difficulty, etc.
You can use a 17M parameter NLI model for simple stuff and get orders of magnitude faster decisions. But the intelligence of that model will be quite a bit weaker.
In some tasks you might be talking about very small differences in accuracy between that model and a 27B parameter LLM decision model for several orders of magnitude in speed. In other tasks, the 17M might be about as good as a random guess.
declanjackson 1 days ago [-]
But isn't there still a meaningful spread of difficulty, risk tolerance, etc. in decision model relevant tasks? Especially at high volumes I think there would still be value in being able to make intelligence/cost/speed/calibration tradeoffs between models
jstummbillig 1 days ago [-]
Any ideas what decider models are actually good? There does not seem to be a strong competitive benchmark culture (as with LLMs) for whatever reason?
(caveat, I work at AA and helped build this Index)
talloaktrees 1 days ago [-]
just found this today and used it as one of the links I sent a coworker who hadn't heard of decision models yet!
iltennico 23 hours ago [-]
[dead]
Oras 10 hours ago [-]
It’s fine considering the cost. I’ve run tests to classify lines from job ads to see which category they belong to, and have to say most of these models are disappointing. I suppose they were trained on long text not short ones.
Based on my tests, the top two (based on quality of classification and confidence level) were OpenAI decision and Liquid D1
stri8ted 1 days ago [-]
It performs well on my local benchmark, and is priced competitively. Good timing too, considering that inception Labs is discontinuing their Mercury models API, which I was previously using. It seems plausible that the bidirectional attention performs better for decision models.
senjin 23 hours ago [-]
Perplexity Decider is my favorite of these new decision models but this version is a step back in my benchmark for my use case. The decision index benchmark[1] actually agrees with this and makes me think I should look at a higher ranked model
I am curious about your use case here, can you elaborate?
neronuser 15 hours ago [-]
Waiting for decision models being available from services like Bedrock. AWS recently published a post where they used decision models to boost evaluation performance, but they had to self-host it or call Jev through a Lambda
redanddead 1 days ago [-]
Complete lack of creativity in the field of AI...
Here's new product, which is just our shitty version of this easily reproducible technology
What is this?
Same for Google and their decisions api
agentdev001 1 days ago [-]
You seem to be confusing market capture for novelty. This almost certainly is not intended to be taken as some greenfield thing from perplexity, same goes for google
jgalt212 1 days ago [-]
Perhaps, but if you're already inside GCP it's nice to have something easy to turn on.
adrianvi 1 days ago [-]
Not just that, if your company approved and gets billed by X cloud, it's quite difficult to request another product from Y cloud
happyopossum 1 days ago [-]
> version of this easily reproducible technology
so you'd prefer indefinite first-mover monopolies?
sowhat1 1 days ago [-]
And all the people involved are making millions to billions. Maybe we're the schmucks.
I don't know how we're going to convince the best and brightest minds of the future to do real work when grifting, corruption, and trend chasing pays the real money. Why try to cure cancer when you can make a guaranteed bag building drones that kill people, gambling apps, or ponzi scheme financial engineering?
throwaway_5753 12 hours ago [-]
V1.3 shows gains over v1.2 and Jev 1.14 on our internal evals for an entity extraction/matching task. Worth scoring it for your use case!
nchmy 23 hours ago [-]
Because of this post, i discovered and dug into various decider and other models. The "locate" mechanism in Decider 1.3 seems to have an equivalent in Clairic from cloudflare and Extract from ScaleDown. I'm particularly impressed with scaledown for my use case - much cheaper and more accurate than the others.
https://www.perplexity.ai/hub/blog/pii-trace-detecting-perso...
https://www.perplexity.ai/hub/blog/a-local-first-agent-for-p...
Here are some thoughts not facts:
On AA Decision Index.
Decider v1.3 and Decider v.1.2 seem to have same benchmarks. However, V1.3 flash beats V1.2 flash by 4 points.
I am not particulary excited about this pattern/trend of having flash versions for decision models. The whole point of Decision models was to deliver faster ouput for high volume requests at much cheaper cost than Reasoning models. Since most reasoning models can and probably do outperform decision models already in accuracy but they are a lot slower and are pricier. Reasoning Models can already return structured ouput.
So your flagship model must be fastest and most accurate.
Unless it can handle some different form of input other than just image and text, I don't see the point of having both flagship and flash version for the same model.
That's rarely the way anything in life works though. Do you prioritize fastest OR most accurate? That's a question that depends on context, task difficulty, etc.
You can use a 17M parameter NLI model for simple stuff and get orders of magnitude faster decisions. But the intelligence of that model will be quite a bit weaker.
In some tasks you might be talking about very small differences in accuracy between that model and a 27B parameter LLM decision model for several orders of magnitude in speed. In other tasks, the 17M might be about as good as a random guess.
(caveat, I work at AA and helped build this Index)
Based on my tests, the top two (based on quality of classification and confidence level) were OpenAI decision and Liquid D1
https://huggingface.co/spaces/multimodalart/jev-decision-ind...
Here's new product, which is just our shitty version of this easily reproducible technology
What is this?
Same for Google and their decisions api
so you'd prefer indefinite first-mover monopolies?
I don't know how we're going to convince the best and brightest minds of the future to do real work when grifting, corruption, and trend chasing pays the real money. Why try to cure cancer when you can make a guaranteed bag building drones that kill people, gambling apps, or ponzi scheme financial engineering?
https://scaledown.ai
https://blog.cloudflare.com/clairic-extraction-model/
https://blog.cloudflare.com/clef-decision-models/
One use case is live content moderation or streaming where reasoning models could be a lot slower and its hard to hardcode logic.