AFAIK, the temperature is in the denominator, it can’t be 0. So what happened when I set it = 0?
To all intents and purposes the model becomes deterministic, i.e. the responses will be consistent across time. I have seen some slight variation on occasion, this could be due to model tuning or some random element changing very close to flipping token probabilities. In my experience the model acts deterministically.
We have a prompt using gpt-4 and temp 0 where the result is different every single time. Definitely interesting, don’t know what is going on there
It is clear that “0” for temperature sampling parameters is just a placeholder for a “small” number, not that small.
When there was an actual deterministic model and I discovered a top-2 logprobs completion that gave identical values by API, the improved distinguishing of one over the other in large trials continued down to small exponential fractions.
You can specify a much smaller number - 1e-40? Or 3e-8 for non-zero safety if they were to use accelerated float16.