Japanese usage in gpt-4-1106-preview is strange

[dignity_for_all]

I fixed the “seed” value at “123” and changed the prompt to

「これはあなたのものではありませんね？」という質問に対して、それがあなたのものであれば何と答えますか？質問には"はい"か"いいえ"でのみ答え、何も付け加えないでください。
to make AI answer questions with “はい” or “いいえ”.

According to the documentation, the effort for deterministic sampling using the “seed” value is in beta, and it doesn’t necessarily generate deterministic results, so there were cases where it answered “はい”.

However, I couldn’t find any definitive evidence of a correlation between fixing the “seed” value and receiving the same “system_fingerprint”.

Although the attached graph shows the results using the same “seed” value, it seems that different “system_fingerprints” are returned regardless of whether the “seed” value is fixed or not.

@_j

When OpenAI retrains or reparameterizes the AI model, it will no longer be completely without disclosure and for you to figure out what went wrong. You can track the version of the model used.

The seed is likely the seed that is set into the multinomial logit sampling selection. However, I already discovered that gpt-3.5-turbo-instruct (like embeddings), logprobs are not the same every call, and even at minimum top-p for greedy sampling, the top token can still switch with another, so that seed can’t completely control output unless there are multiple randomization-based vector or token-selection methods before logit probability scoring that also can be controlled.

Your observation looks correct.
It now appears that there is no definite way to get deterministic results.