Get same_input -> same_output always. force argmax + random seed + clean token queue in generation. what is possible?

[_j]

I don’t work for AI and didn’t program models. We can only extract evidence.

The randomness is obviously not reset to a fixed state between API calls, as this would defeat the purpose of diversity sampling for varied answering, and only ensure the same low-perplexity response each time.

You can replicate this prompt and the next token that it produces (the 46th context element) as one example to probe a case where two top probabilities are almost identical:

Why the API output is inconsistent even after the temperature is set to 0

Read GPT-2 source code. I have. Code self-documents how logits are selected. Temperature increases the probability distance of normalized logits by dividing by a fraction (multiplying the reciprocal of temperature). GPT-3 is just an incremental advance that followed shortly after, except for training on 200x the parameters. GPT-4 temperature obfuscated by the mixture of expert models that has been supposed.

More:

Non deterministic API/Playground GPT-4 responses breaks LangChain ReAct implementation

GPT-4, likely being a mixture of expert models and a synthesis of their results, acts a bit differently than gpt-3xx. One interpretation could be that temperature is scaled differently within specializations, or that there are components that still include multinomial selection. The assertion that a tool is broken even at very low temperature is specious. Let’s play with probabilities: params = {"model": "gpt-4", "max_tokens": 1, "n": 60, "temperature": 0.2, "top_p": 0.99, "messages":…

The latter gives you sample chat endpoint code and gpt-4 results, and we also have the new gpt-3.5-turbo-instruct completion model with raw input that returns logprobs to experiment with.