- They might be dynamically adjusting these at inference time [1]. For example, start with a low temperature and generate samples with increasingly high temperatures until one of them passes some quality gate.
- They don't want you to fine-tune on high temperature completions (rejection fine-tuning). You could call this "rejection fine-tuning rejection".
- SynthID hides the watermark in the sampling RNG. No randomness -> no watermark.
- If you want to distil on the model outputs, you want temp=0 outputs. No temp=0 -> worse distillation.
orbital-decay 9 hours ago [-]
You probably don't want temp 0, especially with Gemini which often fails in the greedy sampling mode in practice, often in ridiculous ways (e.g. multiple thousand token loops). As another comment says it's pretty brittle and this is even reflected in their docs somewhere IIRC. You want the "normal" temperature (whatever it is) and oversampling if necessary.
Certain open models have/had the temperature locked on the official APIs, I assume they just have sampling incompatible with static temperature or do some fancy speculative decoding. It's clearly not to hide anything, as the weights are open and there are always alternative providers.
franky47 14 hours ago [-]
> To improve determinism, define a system instruction with explicit rules for your specific use case.
"Please be deterministic".
dev_dan_2 9 hours ago [-]
> improve determinism
Oh well. I might be to picky here, but how I see things, determinism cannot be improved or worsened, but achieved or not achieved. Or Partially archieved, when analyzing a system that has both components that are deterministic or non deterministic.
There are times to think in absolutes, and when talking about deterministic behavior of technical systems, this is one of them. Join the sith side, we have cookies, and when we say we have cookies, we do.
gregdaniels421 5 hours ago [-]
Probably what is meant is consistency instead of determinism
Zambyte 9 hours ago [-]
In all seriousness, this seems like it could be fine if done well. You can just have a model do a pass over the system prompt and set reasonable parameters based on that. That's probably not what they're doing, but it could be.
rf15 12 hours ago [-]
please oh god please, I don't know how to
aesthesia 21 hours ago [-]
My guess is that RL training being done with particular generation parameters makes models much more brittle to changes in these parameters, and that's why we're seeing changes like this across model providers. But I don't really know.
pixelmelt 21 hours ago [-]
I'm inclined to agree given how unstable Gemma 4 is when not using the "official" sampler settings
19 hours ago [-]
19 hours ago [-]
marcinjachymiak 7 hours ago [-]
It's not surprising since these are all reasoning models now. Token sampling/decoding isn't exposed because the reasoning models are trained around specific configurations. That's why it's been replaced with "reasoning effort" instead (low, medium, high "thinking" levels).
bob1029 14 hours ago [-]
> To improve determinism, define a system instruction with explicit rules for your specific use case.
What if I want to do the other thing? When performing research with many sub agents, having a lot of diversity in the hypotheses is a big deal. If my 5 parallel sub agents all produce the same conclusion I might as well have only ran one.
The latest OAI models have done the same thing. I'm currently adding random variation to prompts to compensate for the lack of higher temperature sampling.
orbital-decay 8 hours ago [-]
Current post-trained models are stochastically and semantically collapsed to a large extent, you'll never get meaningful semantic diversity if your prompt stays the same. This makes the temperature brittle, the model becomes dumb much earlier than meaningfully diverse. There are fancy samplers that claim to overcome this (comments ITT mentioned them), but a) you need a local inference stack to use them and b) I've yet to see any definitive evidence they add any deep semantic diversity without dumbing the model down, most of what I've seen is pretty superficial, including distribution-aware samplers.
>I'm currently adding random variation to prompts
This is better than naive temperature tweaks, if done right.
herpderperator 15 hours ago [-]
I'm curious: If someone wanted to serve models off hardware/silicon directly (like Cerebras or Taalas, and soon Google I think) rather than GPUs, would these parameters still be adjustable at request time? Or would they have to decide that before the model can even start serving and it would be locked in until they reload it (which would make it briefly unavailable)?
StrauXX 7 hours ago [-]
I'd expect that sampling would happen in software. Probably the hardware system would output a vector of probabilities over the tokenspace, just as the nets do when run in software.
I don't know though and am not aware of any docs going into detail here. That being said, sampling is really cheap. So implementing it in hardware wouldn't be worth it.
kouteiheika 20 hours ago [-]
Obligatory "The Conspiracy Against High Temperature Sampling":
wow I wish I knew this post before I installed various UIs with tons of temp/top-X sliders.
They are extremely confusing.
NooneAtAll3 19 hours ago [-]
where can one learn what top_k and top_p mean?
nodja 15 hours ago [-]
The posted answers are either behind a paywall or very obtuse so I'll just explain. I'll assume you know what tokens are.
A models output is not a single token, but a list with the probability for all the tokens that it knows, so we need to use a sampler to select the token that it's going to be the next token in the sentence. For example a simple greedy sampler will choose the token with the highest probability, but samplers normally pick a random token weighted by probability. A model usually knows about ~250 thousand tokens and the probability of some of these tokens are gonna be high, but the vast majority is close to but not actually 0% so there's a chance the sampler might pick some random token that doesn't make much sense, so we filter tokens.
top_k filters the tokens so that only the k top tokens are selected. So top_k=50 will filter those 250k tokens to only 50. This is assuming the list of tokens is sorted by probability.
top_p filters the top tokens until a percentage is accumulated. So if for example if you set the top_p to 0.6 and the model gave the top token a 0.5 (50%) probability and the second top token a 0.2, those 2 token accumulated to 0.7 which is greater than what you set it to (0.6) so no more tokens are selected. If this ran after top_k=50 it'll turn the list of 50 tokens into one of 2.
After each filter parameter is processed, the probability of the tokens is adjusted to sum to 1 (100%), Also note that order of operation here matters, i.e. top_p could be applied before top_k, but most providers follow what's on huggingface, I think I've only seen different implementation in certain local model hosting frameworks.
vintermann 8 hours ago [-]
I used to try to train models ages ago (that is, a couple of years), and I never found out how they got good results from top_k in the first place.
My problem was that as soon as the model had generated an unlikely token, it was quickly steering towards spaces it had never seen during its training. It found itself in unmapped territory, where anything might happen. With high temperatures, it would then generate noisy garbage, with low temperature it would generate repetitive garbage. We tried to fight this with rollout strategies and backtracking, i.e. if you find yourself in a space where all tokens seem equally likely, you're probably in junk land.
Our tricks didn't work very well. But I didn't manage to keep up well enough to learn what worked.
NooneAtAll3 12 hours ago [-]
Thank you. So they are essentially a protection against spurious errors, cool
I don't quite understand the point about order of operations - does it do normalization after every such filter pass? why not leave it to the end?
also: what is top_a? I saw it being mentioned in the GP link
kouteiheika 11 hours ago [-]
> So they are essentially a protection against spurious errors
Only coincidentally. Sampling nonsense tokens will certainly degrade its performance and/or brick it, but it's also there to encourage diversity.
For example, imagine we have the following sentence:
> The color of this ball is ____
Now, what should the model predict for "____"? There isn't really a "correct" answer here. It can be "red", it can be "blue", it can be "green", or any other color. But it's definitely not going to be "ব্যথাя". LLMs output a probability distribution for the next token, so imagine this is the probability distribution that it outputs:
red -> 60%
green -> 19%
blue -> 19%
ব্যথাя -> 2%
So how do we decide which token to pick? Simplest way is to always pick the most probable one (in this case: "red"). In this case we'd ideally want it to be able to output "red", "green" or "blue" (since all of those are reasonable), but never "ব্যথাя" (whose 2% is most certainly noise). So a sampler is essentially an algorithm which lets the inference engine pick the exact token to output from this list.
> I don't quite understand the point about order of operations - does it do normalization after every such filter pass? why not leave it to the end?
Because you can technically compose multiple samplers at the same time in a pipeline, and in some cases their order can matter and give you a different result (or take less/more time to execute). To give you a generic example: imagine you have a list with numbers in random order. You can execute one of two operations on it: (1) sort it, (2) take the leading 10 numbers. If you first sort it and then take 10 leading numbers you'll get a different result than if you'd first took 10 leading numbers and then sorted them.
nodja 11 hours ago [-]
Yes, the whole list will always sum to 1 (100%) because there's lots of more sampling parameters. top_p, top_k and temperature are just the ones that affect output the most. Most parameters do math around assuming the list sums to 1 and order is not always the same, some software even lets you change the order around.
top_a is not very common and is better explained if I explain how the much more common min_p works. min_p filters out tokens below a certain threshold. The formula is <filter threshold> = <min_p> * <top token probability>. So if the top token has 0.5 probability, min_p = 0.1 would cut out tokens below 0.05. This is a tunable that lets you filter out other tokens depending on how confident the model is.
top_a is almost the same formula but you just square the <top token probability>. So <filter threshold> = <top_a> * <top token probability> ^ 2. This makes the filtering ramp up faster (cut out more tokens) if the model has a much more confident top choice, but keep more choices if the model is not so confident.
krapht 19 hours ago [-]
ironically, any frontier LLM will easily generate a tutorial at any detail you like explaining what these are.
if don't have time for that, just know that these are technical parameters that affect how likely it is an llm will produce the same result after being asked the same question.
barkingcat 15 hours ago [-]
Is an llm able to explain to itself what these parameters do, and change its own parameter settings?
chronogram 13 hours ago [-]
An LLM is stateless so every time you send a message a command is run with the parameters, the prompts, and a copy of the chat history when using one of those chat interfaces, and then it returns text. If the output includes explaining it to an LLM that would then be included in the copy of the chat history placed in the new command.
It would be a bit like opening a text editor and typing in that you want to increase the font size. Someone external to the text would have to come along and click the font options.
hehe...
fwiw, the un-editorialized title is "Using the *latest" Gemini models"
I speculate OP wanted to put focus on their chosen detail in the title...
greatgib 44 minutes ago [-]
Indeed, the focus was on this section but it is direct link to the relevant paragraph.
For the "last/latest", it is the result of trying to trim the title enough to enter in the size limited title box and then I didn't notice the typo.
At first I wanted to put the 3.5/3.6 model number because "latest" is not clear and time dependant but it was too long then.
david-gpu 9 hours ago [-]
The OP does not appear to be a native English speaker. And, for example, languages like Spanish don't clearly differentiate between "last" and "latest".
henriquecm8 8 hours ago [-]
I am not a Spanish speaker, but Portuguese is usually similar.
In it we say "último" for last, and "mais recente" for latest, so while there isn't a single word translating, we still have its equivalent.
I imagine it's the same for Spanish.
tolugenius 21 hours ago [-]
> To improve determinism, define a system instruction with explicit rules for your specific use case.
Is this guaranteed to work any better than top_k or top_p? This just sounds like making a smaller version of a Agent.md doc.
janalsncm 19 hours ago [-]
It is guaranteed to work worse than top_k=1, that’s for sure.
furyofantares 18 hours ago [-]
Depends on what you mean by determinism.
sara_mo 19 hours ago [-]
[flagged]
tough 21 hours ago [-]
fwiw sonnet-5 also drops temperature (sonne-4 had it)
kittikitti 6 hours ago [-]
The API endpoints are less useful if you can't even fine-tune these parameters. These endpoints used to be developer oriented and this sort of change would be a huge cause of concern. If you primarily vibe-code, this is not a problem but understand that they're making their product worse on purpose.
greatgib 42 minutes ago [-]
Same thing as the new APIs like assistant API that rip us of the control of the history of messages for the context that is now hidden server side.
temperature, top_p, and top_k are deprecated and ignored. In future model generations, supplying these parameters returns an HTTP 400 error. Remove these parameters from all requests.
charcircuit 18 hours ago [-]
Along with everything else. These parameters can make speculative decoding less accurate increasing the inference cost.
impulser_ 21 hours ago [-]
Good. These have been basically useless for the past few generations of models, and most of the time made the model perform worst.
double2025 14 hours ago [-]
[flagged]
arthiarumugam18 9 hours ago [-]
[flagged]
Rendered at 20:39:51 GMT+0000 (Coordinated Universal Time) with Vercel.
- They might be dynamically adjusting these at inference time [1]. For example, start with a low temperature and generate samples with increasingly high temperatures until one of them passes some quality gate.
- They don't want you to fine-tune on high temperature completions (rejection fine-tuning). You could call this "rejection fine-tuning rejection".
[1] https://rlhfbook.com/c/09-rejection-sampling#related-best-of...
- SynthID hides the watermark in the sampling RNG. No randomness -> no watermark.
- If you want to distil on the model outputs, you want temp=0 outputs. No temp=0 -> worse distillation.
Certain open models have/had the temperature locked on the official APIs, I assume they just have sampling incompatible with static temperature or do some fancy speculative decoding. It's clearly not to hide anything, as the weights are open and there are always alternative providers.
"Please be deterministic".
Oh well. I might be to picky here, but how I see things, determinism cannot be improved or worsened, but achieved or not achieved. Or Partially archieved, when analyzing a system that has both components that are deterministic or non deterministic.
There are times to think in absolutes, and when talking about deterministic behavior of technical systems, this is one of them. Join the sith side, we have cookies, and when we say we have cookies, we do.
What if I want to do the other thing? When performing research with many sub agents, having a lot of diversity in the hypotheses is a big deal. If my 5 parallel sub agents all produce the same conclusion I might as well have only ran one.
The latest OAI models have done the same thing. I'm currently adding random variation to prompts to compensate for the lack of higher temperature sampling.
>I'm currently adding random variation to prompts
This is better than naive temperature tweaks, if done right.
I don't know though and am not aware of any docs going into detail here. That being said, sampling is really cheap. So implementing it in hardware wouldn't be worth it.
https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...
They are extremely confusing.
A models output is not a single token, but a list with the probability for all the tokens that it knows, so we need to use a sampler to select the token that it's going to be the next token in the sentence. For example a simple greedy sampler will choose the token with the highest probability, but samplers normally pick a random token weighted by probability. A model usually knows about ~250 thousand tokens and the probability of some of these tokens are gonna be high, but the vast majority is close to but not actually 0% so there's a chance the sampler might pick some random token that doesn't make much sense, so we filter tokens.
top_k filters the tokens so that only the k top tokens are selected. So top_k=50 will filter those 250k tokens to only 50. This is assuming the list of tokens is sorted by probability.
top_p filters the top tokens until a percentage is accumulated. So if for example if you set the top_p to 0.6 and the model gave the top token a 0.5 (50%) probability and the second top token a 0.2, those 2 token accumulated to 0.7 which is greater than what you set it to (0.6) so no more tokens are selected. If this ran after top_k=50 it'll turn the list of 50 tokens into one of 2.
After each filter parameter is processed, the probability of the tokens is adjusted to sum to 1 (100%), Also note that order of operation here matters, i.e. top_p could be applied before top_k, but most providers follow what's on huggingface, I think I've only seen different implementation in certain local model hosting frameworks.
Our tricks didn't work very well. But I didn't manage to keep up well enough to learn what worked.
I don't quite understand the point about order of operations - does it do normalization after every such filter pass? why not leave it to the end?
also: what is top_a? I saw it being mentioned in the GP link
Only coincidentally. Sampling nonsense tokens will certainly degrade its performance and/or brick it, but it's also there to encourage diversity.
For example, imagine we have the following sentence:
> The color of this ball is ____
Now, what should the model predict for "____"? There isn't really a "correct" answer here. It can be "red", it can be "blue", it can be "green", or any other color. But it's definitely not going to be "ব্যথাя". LLMs output a probability distribution for the next token, so imagine this is the probability distribution that it outputs:
So how do we decide which token to pick? Simplest way is to always pick the most probable one (in this case: "red"). In this case we'd ideally want it to be able to output "red", "green" or "blue" (since all of those are reasonable), but never "ব্যথাя" (whose 2% is most certainly noise). So a sampler is essentially an algorithm which lets the inference engine pick the exact token to output from this list.> I don't quite understand the point about order of operations - does it do normalization after every such filter pass? why not leave it to the end?
Because you can technically compose multiple samplers at the same time in a pipeline, and in some cases their order can matter and give you a different result (or take less/more time to execute). To give you a generic example: imagine you have a list with numbers in random order. You can execute one of two operations on it: (1) sort it, (2) take the leading 10 numbers. If you first sort it and then take 10 leading numbers you'll get a different result than if you'd first took 10 leading numbers and then sorted them.
top_a is not very common and is better explained if I explain how the much more common min_p works. min_p filters out tokens below a certain threshold. The formula is <filter threshold> = <min_p> * <top token probability>. So if the top token has 0.5 probability, min_p = 0.1 would cut out tokens below 0.05. This is a tunable that lets you filter out other tokens depending on how confident the model is.
top_a is almost the same formula but you just square the <top token probability>. So <filter threshold> = <top_a> * <top token probability> ^ 2. This makes the filtering ramp up faster (cut out more tokens) if the model has a much more confident top choice, but keep more choices if the model is not so confident.
if don't have time for that, just know that these are technical parameters that affect how likely it is an llm will produce the same result after being asked the same question.
It would be a bit like opening a text editor and typing in that you want to increase the font size. Someone external to the text would have to come along and click the font options.
I speculate OP wanted to put focus on their chosen detail in the title...
For the "last/latest", it is the result of trying to trim the title enough to enter in the size limited title box and then I didn't notice the typo.
At first I wanted to put the 3.5/3.6 model number because "latest" is not clear and time dependant but it was too long then.
Is this guaranteed to work any better than top_k or top_p? This just sounds like making a smaller version of a Agent.md doc.
temperature, top_p, and top_k are deprecated and ignored. In future model generations, supplying these parameters returns an HTTP 400 error. Remove these parameters from all requests.