Keeping system_msg & req constant, with temperature=0, I get different answers. I got 3 different answers when I last ran this 10 times for instance. The answers are similar in concept, but differ in semantics.
I was expecting the exact same answer every time. Why is this happening?
You may have two words with the same score in the top_p list of word/token choices.
In really simplistic terms, temperature tells it what percent of the top words it can pick the next word from - based in the sum of their probabilities. (This is not 100% correct, but helps explain the next part)
A temperature of zero tells it to pick the top word from the top_p lost. But the list of words could be huge and some words may be at the top of the list with the same probability score - especially short and common words.
What top_p does, is it reduces the list of word choices. So 1 means the list has every word possible, ranked high probability to lowest.
A top_p of 0 basically tells the ai to throw away all the words except the very top ones - so temperature has less to pick from. Even a value of 0.1 will make a big difference
By reducing the number of words on the list, the temperature has less to pick from.
So try a temperature of 0 and a top_p of 0.1 and see if it makes a difference
No harm in trying…
Note: I used “words” in my description above to make it easier to explain - but it is actually “tokens” that are being ranked two different words could start with the same token.
I tried again with top_p = 0.1
Don’t see any difference though.
What you said made sense to me for temp > 0, that it’ll have less options with lower values of top_p. But since at temp=0 it simply picks the highest one, I don’t understand how changing top_p would have a big difference in this case.
I’m still seeing 3 different answers if I run it 10 times.
GPT generation process is non-deterministic by default. You can see a more thorough discussion on this on this thread (and its associated link). TL;DR: the problem is that the token with “highest probability” is ill-defined due to the finite number of digits that you’re using for multiplying probs and storing them. Hope it helps