Achieving deterministic API output on language models - HOWTO

[_j]

Yes, it turns out that the 3.5-turbo models, while indeed my small top_p setting locks in the top token more than actually setting to 0, there is still non-determinism in the logits, and there can be position changes of “top” in long outputs.

I did a thorough investigation here, using the gpt-3.5-turbo-instruct model:

Feature Request: Deterministic Answer Option for Unit Testing

I have investigated the problem that causes non-determinism. The source or root is with the model vectors and logits produced. Their significant figure changes between runs are similar to embeddings. When the values of likelihood of a token appearing every run continuously change by up to several percent, the top-ranking probability that comes from the language model can change. Here, as part of my investigation with many different styles of API calls and scripts to process the data returned, …