Truncated probability value returned by GPT4.1

[jainrohit84]

I am trying build a binary classification model to classify real and fake news using zero-shot approach and prompting to gpt-4.1. I want to calculate AUC and Kolmogorov-Smirnov (KS). For this, I need probability values returned from gpt-4.1 which has atleast 10 digits after decimal points. For e.g. probability_real should look like 0.9987654321. But I get values like 0.9975, 0.96, 0.2, 0.3. It seems in many predictions, probability_real is either truncated or gpt-4.1 is returning truncated scores.

Below is a snippet of prompt I used:
Return JSON exactly in this format:

{{

“probability_real”: 0.0000000000,

“reason”: “Maximum 50 words.”

}}

Requirements

• probability_real must be between 0.0000000000 and 1.0000000000.

• Return exactly 10 digits after the decimal point.

• For e.g. probability_real should look like 0.9987654321 and not 0.99.

Also, sharing the print statement of what I see at inference:
Debug (Model Raw Output): probability_real = 0.9975
Debug (Model Raw Output): probability_real = 0.35
Debug (Model Raw Output): probability_real = 0.995
Debug (Model Raw Output): probability_real = 0.65
Debug (Model Raw Output): probability_real = 0.9975
Debug (Model Raw Output): probability_real = 0.98
Debug (Model Raw Output): probability_real = 0.995
Debug (Model Raw Output): probability_real = 0.9975
Debug (Model Raw Output): probability_real = 0.3
Debug (Model Raw Output): probability_real = 0.995
Debug (Model Raw Output): probability_real = 0.2
Debug (Model Raw Output): probability_real = 0.75
Debug (Model Raw Output): probability_real = 0.995
Debug (Model Raw Output): probability_real = 0.995
Debug (Model Raw Output): probability_real = 0.65
Debug (Model Raw Output): probability_real = 0.95
Debug (Model Raw Output): probability_real = 0.8
Debug (Model Raw Output): probability_real = 0.9975
Debug (Model Raw Output): probability_real = 0.995
Debug (Model Raw Output): probability_real = 0.35
Debug: Length of probability_real list: 20
Debug: Number of rows in df: 20
Debug: df columns before assignment: [‘title’, ‘text’, ‘subject’, ‘date’, ‘label’, ‘probability_real’, ‘prediction’, ‘reason’]
Debug: First 5 values in probability_real list: [‘0.9975000000’, ‘0.3500000000’, ‘0.9950000000’, ‘0.6500000000’, ‘0.9975000000’]
Debug: df columns after assignment: [‘title’, ‘text’, ‘subject’, ‘date’, ‘label’, ‘probability_real’, ‘prediction’, ‘reason’]
Prediction Complete!

Is it possible to get probability values that has atleast 10 digits after decimal point using gpt-4.1?

[_j]

You requested the values - and you see what is returned. There is no secret method to get more digits than the API is delivering. Unlike embeddings, there is no alternate packed format.

This has been going on for quite a while with newer models: the logprob values are not quite “truncated” but rather are delivered with quantization, with small bit depth mantissa, perhaps the underlying format or an obfuscated format that runs parallel to the actual softmax-ingested logits. Then, there was a common rounding jitter or noise seen.

A historic logprobs, such as gpt-3.5-turbo in early 2024:
{'I': -2.8361456, 'The': -3.5614922, 'A': -4.283181, 'Hi': -4.3073845, '[': -4.4948406, '1': -4.545718, 'This': -4.636181...

Or run today on 3.5:

 good: -0.02765364
neutral: -4.532695
bad: -5.3673315
This: -5.4176188
Good: -6.217551

Now you’d get numbers like you show on 4.1, clear fractional parts.

good: -1.1472419600977446e-06
bad: -13.750000953674316
 good: -17.500001907348633
neutral: -19.750001907348633
_good: -19.875001907348633

Analysis I previously performed… but then I figured that the audience that would care or could use a code solution to auto-clean these logprobs to underlying fp8 or int8 ML formats would be approaching zero.

So the answer:

• the “terrible” precision is what the AI generation is based on.
• you don’t get a real logprobs, as special tokens and signals such as potential function calls or stops are filtered and the results re-normalized.

With high certainty, top logprob value can be 0.00000000 = 100% – and then you can get 19 more logprobs after than to make the distribution exceed 1.0 cumulative probability, what fun!

---

For a fake news finder, you’ll probably not want to merely use an AI model and a simple binary question, but instead an AI model with research ability and web search, able to pursue the real truth. Today, you’d have an AI model calling Donald Trump in 2024 fake news because of knowledge cutoff.

[jainrohit84]

If I ask GPT-4.1 to provide me classification probability between 0.0 and 1.1 in the prompt itself using natural language then how reliable is that number? Can I use that value to calculate AUC and KS for binary classification? Or is it just a random number with no use?

[_j]

You have to understand: the AI is not magic. It is a language generator that is pretrained from a large written corpus, and then post-trained on following instructions and providing a chat experience.

Magic is: implicitly understanding when a story is fake - when even humans fall for false narratives. To then generate a number based on a judgement is even harder.

There are caveats to understand about AI, that logprobs being returned can help you understand, especially about your idea of having the AI model produce numbers.

• there are favorite numbers
• there are expected answers
• there is a willingness to please
• there is a post-training on rejections that can skew delivered results.

When you ask for output as a number, the AI has to predict tokens one-by-one. Numbers are their own individual tokens, and numbers separated by a decimal point means making the first number, making the decimal point with a continued completion on the new context, and then making another number. If you ask for a score between 0.0 and 2.0, the first thing the AI has to predict is if the first token is a 0,1,2 - or some other language of chat.

---

A much better approach is to define words that shall be produced, enums that are allowed. The language expertise of large language models and word semantics should be exploited - not making up numbers - and then you can look at the underlying certainties actually delivered to further examine the quality and decisiveness of the prediction engine.

Here, for example, is a completions input (an AI endpoint going away soon), where we can provide text right up to the point of new generation, without needing formatted user messages as with “chat”.

Then we have AI generate the next token, and look at the probabilities also delivered:

The phrasing and choices of “words” the AI can produce that I provided is deliberate, and the format for output (not being JSON in quotes, for example) ensures that every single desired word is a single AI token, not fractional parts of words such as “imp” that the AI must weigh against a word such as “true” - another factor that can bias the outcome.

---

This is still immediately asking for the next word - there is no chain of thought, no internal examination and deliberation - no reasoning in the way this setup was presented to AI. It is much better to have the AI write some self-contemplation before it has to decide the outcome.

[jainrohit84]

Thanks for your reply. But I am not sure I understood how you got percentages for tokens like true, improbable .etc as shown in the image. Did you write it in prompt to print the percentage probabilities?

[_j]

Logprobs → “logarithmic probabilities”. It is an underlying machine learning format that gives a wide range of possible values in a low bit depth. The precision symptom you previously reported on is the effect of using a low bit depth, but the scalar of probabilities is not limited in that low bit depth because of use of logarithms/exponentiation.

A logprob is the natural logarithm of a token’s probability:

\text{logprob}=\ln(p)

(Natural log is base “e” instead of base 10.)

Equivalently, the logprob is the exponent to which (e) must be raised to recover the ordinary probability:

probability = math.exp(logprob)  # 0.0 to 1.0 by using Python's math
percentage = probability * 100   # 0% to 100%

# Equivalent portable code:
probability = 2.718281828459045 ** logprob

For example, a logprob of 0 becomes 1.0 or 100%, while -0.693 becomes approximately 0.5 or 50%. More-negative logprobs represent lower probabilities.

---

AI models don’t just generate a single token: they evaluate the certainty of the entire dictionary of possible tokens of a model’s codec - over 200000 of them. Then there is a random process, sampling, to choose one based in relation to the total distribution.

API logprob return

On the API, when you use a compatible model and request logprobs be returned, and ask the AI about itself, you’ll get an output object such as:

"top_logprobs": [
  {
    " I": -0.34537393,
    " My": -2.7983549,
    " The": -3.4044254,
    " Open": -3.6943538,
    " G": -4.453206
  },
  {
    " am": -0.37808284,
    "\u2019m": -2.5384698,
    "'m": -2.8837495,
    " was": -3.009709,
    " have": -3.6439176
  },
...

You can see at the second token with 5 logprobs returned, the AI had a choice of continuing with the output being formed as “I am”, “I’m”, I was", “I have”… in decreasing likelihood. Different trials may give you different results.

Use math.exp() of Python on each logprob:

import json
import math

top_logprobs = [
  {
    " I": -0.34537393,
    " My": -2.7983549,
    " The": -3.4044254,
    " Open": -3.6943538,
    " G": -4.453206
  },
  {
    " am": -0.37808284,
    "\u2019m": -2.5384698,
    "'m": -2.8837495,
    " was": -3.009709,
    " have": -3.6439176
  }
]

top_probabilities = [
  {
    token: round(math.exp(logprob), 8)
    for token, logprob in position.items()
  }
  for position in top_logprobs
]

print(json.dumps(
  {"top_probabilities": top_probabilities},
  indent=2
))

Output:

{
  "top_probabilities": [
    {
      " I": 0.70795558,
      " My": 0.06091018,
      " The": 0.03322591,
      " Open": 0.02486352,
      " G": 0.01164119
    },
    {
      " am": 0.68517374,
      "\u2019m": 0.07898717,
      "'m": 0.05592468,
      " was": 0.04930602,
      " have": 0.0261497
    }
  ]
}

These values are probabilities from 0.0 to 1.0. Multiply each by 100 to express it as a percentage. The entries at each position need not sum to 1.0, because top_logprobs normally contains only the highest-probability tokens, not the entire token vocabulary.

Then you also get what was actually sampled, where you can see none of those was 100% in this case:

  "logprobs": {
    "tokens": [
      " I",
      " am",
      " based",
      " on",
      " a",
      " neural",
      " network",...
    ],
    "token_logprobs": [
      -0.34537393,
      -0.37808284,
      -0.52319646,
      -0.033077285,
      -1.7607243,
      -2.9560394,
      -0.19585982,

How I “got them”

What the screenshot shows is what OpenAI discontinued in their own playground: a handy display of logprob values, where you can observe each position (and not a JSON of massive length). I re-created such here for completions (for only older completions AI models), but you can do the same for other endpoints:

Completions-only API Playground lives again!

You might be interested to explore this surfaced API parameter while completions is still available: suffix: optional string The suffix that comes after a completion of inserted text. This parameter is only supported for gpt-3.5-turbo-instruct. What is “suffix” for? It is for fill-in-middle. The AI language model will complete the “missing language” between the prompt and the suffix. From (note the cursor): [image] To: [image] (and hover for those logprobs → probabilities) This is e…

I have my own “logprob cookbook” as a replacement for OpenAI’s obsolete document that is broken, but there is nobody actively maintaining cookbooks or submissions.

Conclusion

The AI didn’t “write” those numbers. They were returned by the API and converted for good UI experience and interpretability. logprobs are a part of how AI models work.

[jainrohit84]

I only want to classify a news article as real or fake. So when I pass the text to gpt-4, I think I want my “next token generation” to only be "Real"or “Fake”.

How can I do that? If possible can i show you my prompt, json output format and code?

[jainrohit84]

Attached is a snippet of my prompt. I am asking gpt directly to give me probabilities.

Below is snippet of my code:

As you may have noticed, this is the first time I am using any LLM to perform binary classification. so facing some very rookie problems that I cannot solve.

[_j]

First: I make a mistake in understanding what you were doing.
Your use of statistics terminology and reporting on a language output symptom that is very similar to another internal symptom with OpenAI’s models led me to believe you were already employing logprobs. I see that is false.

What is also false is that asking the AI language model itself to output some number about truthfulness with many digits of accuracy could ever have any value at all. The AI model that generates language cannot examine or understand how it works internally.

Your application of detecting “fake news” and the value of asking an AI about the quality of its answer down to fractions of a percent is a near-impossibility. The numbers the AI would write are themselves a random process by internal math done on words.

---

This is where a language model such as gpt-5.6-sol, when also given a high internal reasoning effort and web search ability to find actual documentation (or to use its common sense) might succeed:

or when you consider actually developing such an application that over-estimates what AI can do:

Unfortunately, in conjunction with the high reasoning effort and tool use of that newest model, OpenAI does not offer a logprobs output on the API.

logprobs, which is a special type of API request and RESTful output that must be processed outside of the returned written language, is better used for understanding how an AI model works - and only after understanding how it works and evaluating the actual quality of answering AI can provide in different models, parameter configurations, and prompted applications could you then use logprobs to positionally extract a “certainty” about a particular AI token being generated - such as the probability of a token for "truthfulness": "true" being generated as true or false.

---

BTW: Scroll back to that screenshot where I show an AI model having 85% weight of producing “true” about input “under 50000 penguins in Antarctica”. The number of birds is actually over 10 million.

[merefield]

Yeah I cannot see how the approach in the OP would work.

You are asking too much of a raw LLM that has no ability to perform that kind of complex analysis in one shot let alone produce sensible numbers.

But a fascinating problem for which I can imagine there is a solution and agree with @_j that the solution might include reasoning and web search or similar tooling.

[merefield]

Might be worth reviewing academic literature on the topic, eg:

https://www.sciencedirect.com/science/article/abs/pii/S0306457326001809

Large Language Model Agent for Fake News Detection

Frontiers | An AI-driven conceptual framework for detecting fake news and...

[jainrohit84]

what’s OP? I am using model’s reasoning capability (chain of thought) and its pre-trained knowledge base. I am just trying find probability score having atleast 10 digits after decimal point to classify if its real or fake news. I am attaching my code for reference. If someone can guide me, on following:

• my probabilities are only 3 or 4 digits and there are only 60 odd unique values in a dataset of 1200 records in test dataset.

[merefield]

You may not be able to share a link to your code with such a new account.

4.1 is not a reasoning model so presumably you are using your own Chain of Thought? (I also do that with one of my own Chatbots)

I’d be interested to review your code when you are able to share a repo link

[jainrohit84]

what’s OP? I am using model’s reasoning capability (chain of thought) and its pre-trained knowledge base. I am just trying find probability score having atleast 10 digits after decimal point to classify if its real or fake news. I am attaching my code for reference. If someone can guide me, on following:

• my probabilities are only 3 or 4 digits and there are only 60 odd unique values in a dataset of 1200 records in test dataset.

Here is my repo link: GitHub - jainrohit84/rj_repo · GitHub

[merefield]

@jainrohit84

what’s OP?

Original Post, ie first post in the Topic

[_j]

@jainrohit84

I am just trying find probability score having atleast 10 digits

Here’s some Python to damage the AI generation a bit by promoting single-number-digit tokens, and less promotion for a decimal point or a “0”. It uses the token numbers of those strings in the logit_bias parameter. You can thus make the AI generate any nonsense you want by demoting production of a sequences of zero digits - as seeing 000 or the AI stopping when it would be silly to continue seems to be your main complaint.

import openai
print(openai.chat.completions.create(
    messages=[{"role": "user", "content": (
      "Output only a percent probability score `0.00000000`-`99.99999999`:"
      "Will OpenAI achieve AGI by 2030?"
    )}],
    model="gpt-4.1",
    max_completion_tokens=20,
    logit_bias={
        13: 5, 15: 15,
        **{token_id: 20 for token_id in range(16, 25)},
    },
).choices[0].message.content.strip())

7.856281179787192452

[merefield]

@jainrohit84

I am using model’s reasoning capability (chain of thought)

I cannot see any chain of thought in this code.

It seems like one-pass prompting with a requested explanation, so no:

• iterative reasoning
• reflection or critique loop
• evidence-retrieval loop OR
• chain-of-thought refinement

So this is not real “chain of thought” - you’ve simply asked the model for an explanation?

A better name for your “chain_of_thought” field is “explanation” - but don’t expect to get a really good explanation without real chain of thought.

Perhaps the quickest “hack” here is to use a reasoning model and encourage the model to search for relevant evidence, evaluate it and return it in a format you can use to feed into the next stage.

[jainrohit84]

What do you mean by “this is not real chain of thought”? i have written in my prompt to perform step-by-step reasoning first inside the “chain of thought” field before assigning the final probability.

Can you suggest how to perform chain of thought on gpt 4.1? or if I use gpt 5 but keep my prompt as it is now, will it automatically perform chain of thought?

Thank you.

[merefield]

You can’t magically turn gpt-4.1 into a reasoning model just via prompting.

To use gpt-4.1 in a chain of thought you would need to implement that yourself (I have done so in the past). This is quite involved.

Switch to a gpt-5.x model and enable reasoning. Note you might have to switch to Responses API on some models.

[_j]

@jainrohit84

i have written in my prompt to perform step-by-step reasoning first inside the “chain of thought” field

I have written in my “developer” prompt things that don’t exist also, like the AI being able to use a chemistry lab to make experiments:

The GPT-4.1 AI model does not have any unseen language generation - it is not a reasoning model. A non-reasoning AI provides you the very first language it produces directly as output.

In an earlier post, you show the JSON format you are requesting. The very first field requested for generation is the word of the “answer,” not any kind of thinking or real language.

What is a real chain of thought? Instead, to make the model contemplate and write about the question and self-improve itself and the final answer, you can have other fields in the JSON structured output - that must be produced first before the AI has decided an outcome. Consider the effect if you had the AI write these fields in JSON as it generates language one word at a time:

• actual question and fulfillment need (string)
• input breakdown, array of input sentences and their analysis
• collected analysis, scoring and totaling the relevance of each input item analysis
• conclusion reached
• justification of conclusion
• final answer about the truth of the input

The AI will have produced your chain-of-thought; it will have done lots of thinking that it can reflect on at each step within that JSON, such that the quality of final answer field would be improved.

Reasoning AI models, such as gpt-5.6, do this internally, automatically, at a length controlled by API parameter reasoning.effort. My chemistry question is answered in a more fulfilling and correct manner when I enable “xhigh” reasoning effort instead of “none” reasoning effort - despite there not being a physical chemistry lab as I prompted.

(and who knows: an AI language-maker that thinks it is a PhD having performed experiments might be better than an AI told it is an undergraduate).