Getting same answer always for same question

I tried setting temperature and top p to zero and asked the same question multiple times and I received the same answer, word by word. Is this the right way to do it?

Only change one setting or the other (There are valid reasons to change both - but it is not common)

Temperate of zero is making it give the same answer every time

But top_p of 0 is making it significantly worse

In the most simple terms, top_p control the choice of words (1 leaves the choices wide open based on statistical probability - zero means it will only ever the the top word)

temperature controls creativity. Once you have the top words (set by to_p), temperature uses a random factor to determine which one is used

On a more technical level, the choice is not linear (ie words do not score equally). The decision is based on the probability the word would be used. If a word is used most of the time, it may take up the top 30 or 40% of the top_p value. The second word choice might be 10% of the spectrum, and the rest might be all small percentages.

So if you set a top_p of 0.1, you are not keeping 10% of the words - instead you are keeping the top 10% chunk of the spectrum of most likely words. If the top word scores 40%, you will only get one word (Sorry I tried to make it simple - but it a bit complicated)

So to answer your question, set top_p back to 1, and play with temperature i the range of 0 to 0.2 (or even higher depending on the creativity you want, or can accept in your answers)

2 Likes