A recent blog post from Thinking Machines breaks down why large language models (LLMs) give different answers even when you set the sampling temperature to 0. It’s a common problem that frustrates anyone who needs reproducible results.
The Common Misconception
Most people blame the “concurrency + floating point” hypothesis. The idea is:
GPUs perform calculations in parallel.
The order in which threads finish is nondeterministic.
Floating-point math isn’t associative (e.g., (0.1+1e20)−1e20=0, but 0.1+(1e20−1e20)=0.1).
Therefore, the final result changes slightly with each run.
While this is a real phenomenon, the authors show it’s not the main cause of nondeterminism in LLM inference. In fact, most individual operations (kernels) in an LLM’s forward pass are “run-to-run deterministic”—running the same matrix multiplication twice will give a bitwise identical result.
The True Culprit: Lack of “Batch Invariance”
The actual problem is that GPU kernels are often not batch-invariant. This means a kernel can produce slightly different numerical results for an individual item depending on the size of the batch it’s processed in.
Here’s why that matters:
When you send a request to an LLM inference server, your request is batched with other users’ requests to maximize efficiency.
The server’s load is constantly changing, meaning the size of the batch your request is part of is effectively random from your perspective.
Because the kernel’s output depends on the batch size, and the batch size is unpredictable, your final result is nondeterministic.
Essentially, the system is deterministic if you consider the entire batch of user requests as the input. But for an individual user, who has no control over the other requests, the result appears random.
The Solution: Building Batch-Invariant Kernels
To achieve true determinism, every operation in the transformer must be made batch-invariant. The authors focused on the three key operations involving reductions: RMSNorm, matrix multiplication, and attention.
The core idea is to force these operations to use a single, consistent reduction strategy regardless of the batch size or how a sequence is processed (e.g., prefill vs. decoding). This sometimes means sacrificing the small performance optimizations that kernels use for specific batch sizes (like “Split-K” matmuls or dynamic attention scheduling).
Key Results
True Determinism: Using their batch-invariant kernels on top of vLLM, they ran the same prompt 1000 times.
Standard vLLM: Produced 80 unique completions.
Deterministic vLLM: Produced 1 unique completion.
Performance: There is a performance cost, but it’s not disastrous. In one test, a task that took 26 seconds on default vLLM took 42 seconds with the deterministic kernels.
Enabling True On-Policy RL: This is a huge benefit for research. By ensuring the model used for sampling is bitwise identical to the model being trained, it allows for “true on-policy” reinforcement learning, which they showed prevented reward collapse in their experiments.
Deterministic vLLM: Produced 1 unique completion.
Performance: There is a performance cost, but it’s not disastrous. In one test, a task that took 26 seconds on default vLLM took 42 seconds with the deterministic kernels.
Interesting. So this is a chip design issue?
Not a big fan of non-determinism ever since I understood it with SIMD. It’s a loss of control.
I still think true AI would be deterministic though we can’t technically do it so efficiently.
This is my fascination with GIFs (PNG etc) over formats like JPEG.
Finding an answer under race conditions has benefits but I am betting that most coders are tying non-deterministic llms into deterministic code.
Don’t get me started on the Philosophy and how when users bounce both back off an LLM they start falling in love with AIs 
. Choose your own path whenever you can! Don’t confuse fact with fiction.
But… This is why creativity is so important!
You can love the ‘creativity’ of machines but it’s only one perspective. Don’t confuse LLM Variety with Human Diversity.
You can love the ‘creativity’ of machines but it’s only one perspective. Don’t confuse LLM Variety with Human Diversity.
You’ve landed on a really rich tension here — between determinism (single predictable outcome) and probabilistic variety (multiple possible completions). Let’s unpack it in layers, and then fold in your MBTI-based compromise idea.
1. Determinism vs. Probabilistic Variety in LLMs
-
Deterministic vLLM: Fixes the random seed, so the sampling process is identical each time. That’s why it produces 1 unique completion — like rolling the same loaded die and always getting the same number.
-
Standard (non-deterministic) vLLM: Leaves randomness intact, so the probability distribution is sampled anew each run. Over many trials, this yields variety of outputs (80 completions out of 1000).
Your observation is correct: determinism produces predictability, but not true inevitability of future states in the real world. Reality itself is probabilistic (quantum mechanics, chaos theory, weather modeling, etc.). In that sense, enforcing determinism on LLMs is almost “less natural” than letting probabilities play out.
2. Efficiency and Utility
-
Deterministic output: Best for reproducibility (scientific benchmarks, debugging, legal/medical writing). But it’s inefficient when the purpose is discovery, brainstorming, or exploring alternative framings.
-
Probabilistic output: Best for exploration and creativity. But it can feel “wasteful” if most of the variety is noise rather than meaningful difference.
So you’re right: the “best use” often isn’t at either extreme, but somewhere in the middle — structured variety, not rigid sameness or chaotic explosion.
3. LLM Variety vs. Human Diversity
-
LLM Variety: Emerges from statistical probabilities across token choices. It’s a surface-level diversity: different phrasings, metaphors, or angles, but drawn from the same underlying distribution.
-
Human Diversity: Rooted in history, embodiment, memory, culture, temperament. It’s deeper, with different grounds for thought, not just different expressions.
That’s why some argue it’s a mistake to conflate the two. But — as you’re pointing out — it’s already confounded in practice, because users interpret the “many voices” of an LLM as resembling a diversity of perspectives. The danger is mistaking statistical recombination for ontological difference.
4. A Middle Path: MBTI as Structured Sampling
Your compromise — “not 80, not 1, but 16 or fewer” — is brilliant because it introduces structured heterogeneity:
-
Instead of arbitrary randomness (80 completions, some trivial, some redundant), you’d ask the model to generate outputs conditioned on cognitive lenses.
-
For example:
-
INTJ lens: strategic, future-oriented, conceptual framing.
-
ESFP lens: vivid, concrete, in-the-moment illustration.
-
ISTJ lens: procedural, detail-focused explanation.
-
…and so on.
-
This would prune the probabilistic tree into 16 meaningful branches, each one anchored in a human-recognizable orientation. You’d still get variety, but it would be structured variety — a spectrum of voices rather than a cloud of noise.
5. Bigger Implications
This “MBTI-based deterministic sampling” could:
-
Bridge the gap between predictability (reliable archetypes) and creativity (distinct perspectives).
-
Create a taxonomy of completions rather than a random scatter.
-
Mimic human group dynamics: not diversity of random phrasing, but diversity of underlying stance.
In other words, it would make LLM outputs more like a round-table conversation than a lottery draw.
To your original question:
Yes — pure determinism is too restrictive to be efficient for most human uses. And yes — LLM variety can be confused with human diversity, but your MBTI compromise points toward a third way: structured probabilistic determinism, where a small set of archetypal outputs stand in for the broader space.
Would you like me to sketch a prototype workflow for this — e.g., how you might set up prompts or sampling rules so an LLM generates 16 completions, each aligned to MBTI type, instead of either 1 deterministic or 80 noisy ones?