GPT-4-Turbo consistently getting a number wrong

API
1

I’ve come across a scenario where I’m trying to extract key details from candidates interview notes. Using the latest GPT-4-Turbo preview model available on Azure OpenAI & function calling to describe as a JSON schema the notes I want pulled out.

The notes contain a reference to growing revenue from $25 - $50 million over the time period.

The output of the function call response consistently contains a reference to this, but says $24 - $50 million instead.

It does this every time.

I’ve even added a “citation” property to the JSON schema and updated the prompt with instructions to include a snippet of text extracted exactly from the notes that supports it’s notes, and in that property it extracts the correct sentence but with the $25 changed to $24 in that reference as well.

I’ve rarely seen a “mistake” like this in extraction, and never one that is wrong in the exact same way so consistently.

Has anyone encountered a similar case? If so, what (if anything) helped?

2

Welcome to the community!

That does look pretty interesting. Do you think you can share a prompt where this happens?

Which exact model are you using?

3

The numbers 0 (or 00 or 000) to 999 are their own unique tokens.

You can retrieve the logprob values for different inputs and see why “repeat back 25” has a high chance of producing 24 in this particular case.

Then a negative logit_bias could affect the generation and unbias this bad generation… or not if using the “tool call” API and its enabling unwanted “JSON mode” with its anti-developer ignoring of logit_bias.