Great question! You report your self-discovery that even with the same prompt, AI language models can produce different outputs on separate runs. In other replies, you are getting some “how you can fix this” answers, but really you should be getting, “enjoy that design choice” answers.
This isn’t a bug, but a feature stemming from the core of how these large language models generate text. Unlike a simple calculator that always gives the same answer for the same input, GPT-based AI is designed to be creative and generate diverse, human-like text. Let’s explore why.
Imagine you’re asked to complete the sentence, “The cat sat on the…”. There are many valid completions: “mat,” “chair,” “fence,” and so on. A deterministic model would always choose the single-most likely word, perhaps “mat.” However, a large language model AI considers a vast range of possibilities a word at a time (actually, a token at a time, which can be parts of words), each with its own probability. Instead of always picking the most probable word, it uses a probabilistic approach, sampling from this distribution of possibilities of all tokens possible to generate. This is why you get different, yet often valid and interesting, completions each time you run the model. This probabilistic approach, based on sampling from a cumulative probability distribution, is key to the model’s ability to generate creative and nuanced text.
In using ChatGPT, you’ll see high diversity. OpenAI is often testing slightly different models, either obvously or silently, to gather feedback, but that is not the only reason for each output being different. It is the very basic fact that there is a mechanism to give you diverse possible choices of outputs, and for you to upvote, and which would generate more training data.
Now, let’s dive deeper into the technical details of this sampling mechanism, and explore how settings that you can send to the API influence the text generated.
Why Cumulative Probability Sampling?
At the heart of these models lies the prediction of the next token in a sequence. The model outputs a probability distribution over its entire vocabulary for each possible next token. Instead of always picking the highest probability token (greedy decoding), which can otherwise lead to repetitive and predictable text. OpenAI will employ cumulative probability sampling, also known as nucleus sampling, to tweak how much low-likelihood output is seen, or can just have their models completely sampled from the raw prediction values. This method allows us to inject creativity and diversity into the generated text. Think of it like choosing a path through a branching tree of possibilities. Greedy decoding always takes the most obvious path, while cumulative probability sampling explores less-trodden paths, leading to more surprising and interesting results, while still favoring that more good outputs will still show up.
This sampling approach is crucial for several reasons:
-
Avoiding Repetition: Greedy decoding often gets stuck in loops, repeating phrases or patterns. Sampling breaks free from these loops, generating more varied and natural-sounding text.
-
Generating Creative Text: For tasks like storytelling or poetry generation, sampling is essential for producing imaginative and engaging content. It allows the model to explore different stylistic choices and come up with novel combinations of words.
-
Reflecting Uncertainty: The model’s predictions aren’t always certain. Sampling acknowledges this uncertainty by sometimes choosing less probable but potentially more interesting tokens. This can lead to more nuanced and human-like text.
Technical Underpinnings of Sampling from Softmax
The model’s output is a softmax distribution – a probability distribution where each token in the vocabulary is assigned a probability between 0 and 1, and the sum of all probabilities equals 1. Cumulative probability sampling works by:
-
Calculating Cumulative Probabilities: We sort the tokens in descending order of probability and calculate the cumulative probability for each token. The cumulative probability of a token is the sum of its probability and the probabilities of all tokens preceding it in the sorted list.
-
Generating a Random Number: We generate a random number between 0 and 1.
-
Selecting the Token: We select the token whose cumulative probability is the first to exceed the random number. This ensures that tokens with higher probabilities are more likely to be selected, but lower probability tokens still have a chance.
Controlling Sampling with top_p (Nucleus Sampling) and Temperature
-
top_p (Nucleus Sampling): This parameter controls the size of the “nucleus” from which tokens are sampled. For example, top_p = 0.9 means that only the tokens comprising the top 90% of the probability mass are considered. This helps to filter out very low probability tokens, which are often nonsensical, while still allowing for some diversity.
-
Temperature: This parameter controls the “sharpness” of the probability distribution. A higher temperature (e.g., 1.0) makes the distribution flatter, increasing the probability of selecting less likely tokens and leading to more diverse, but potentially less coherent, text. A lower temperature (e.g., 0.2) makes the distribution sharper, concentrating probability mass on the most likely tokens, resulting in more predictable but often higher quality text.
These are handles which the API has historically exposed on non-reasoning models.
To sum up
The same inputs can generate different outputs - by design. This is to break up potential looping repetitions and also make the output seem more human.
but I want to learn how experienced users do it.
