Leading whitespace missing in GPT4 response

API
1

Hello.
I have recently encounter an issue with the API and GPT4, where it does not generate a leading white space when appropriate.

A trivial example:

text = “The capital of France is”
message = [
{“role”: “user”, “content”: text}
]

result = openai.ChatCompletion.create(
model=“gpt-4”,
messages = message,
max_tokens=50,
top_p=1,
temperature=0.0,
stop=[“.”, “\n”])

print(result)

The output is “Paris” with no leading space.

…
“model”: “gpt-4-0613”,
“choices”: [
{
“index”: 0,
“message”: {
“role”: “assistant”,
“content”: “Paris”
},
“logprobs”: null,
“finish_reason”: “stop”
}
…

Compare this to the response for completion with 3.5-turbo-instruct:

result= openai.Completion.create(
engine=“gpt-3.5-turbo-instruct”,
prompt=text,
max_tokens=50,
top_p=1,
temperature=0.0,
stop=[“.”, “\n”])

print(result)

The output is " Paris" with a leading space.

…
“model”: “gpt-3.5-turbo-instruct”,
“choices”: [
{
“text”: " Paris",
“index”: 0,
“logprobs”: null,
“finish_reason”: “stop”
}
],
…

For my purposes I cannot know if a space needs to be appended for the completion. I am performing a beam search using top_logprobs and the completion may start in the middle of a word (e.g., “In France, the capit” the response should be “al is Paris”).

Is this due to the nature of chat completions? Will it never generate a leading whitespace?

Thanks!

2

I think we can chalk this up to the difference between the chat/completions endpoint and the completions endpoint.

The key element being chat.

In the chat/completions endpoint, the paradigm is two entities having a conversation. It would be unusual if you texted me “the capital of France is” and I responded " Paris".

Whereas the paradigm in completions is a single, uninterrupted stream of text that is continually appended to. So it makes sense to add the missing space.

1 Like
3

Thanks, yeah that make sense for a chat.
For each chat response, we were trying to do a expansion of a tree of responses using top_logprobs and repeated API calls to the chat endpoint.

So looks like either we will have to add logic to determine if a space is needed or instead continue to rely on the completions endpoint and pre-GPT4 models.

4

A few of us are actively asking for a GPT-4-turbo-instruct… No word on whether we’ll get it or not.

2 Likes
5

Ah yeah, that would be great if they made a GPT-4 instruct model!

1 Like
6

Simply, the chat models cannot complete text naturally.

Messages are contained within containers of special tokens.

Additionally, there is a “prompt” added at the end of your input messages, the start of a new message where the AI will write

<special_token>assistant<special_token>(complete here)

So it is seen as the equivalent of writing at a new line, and starting a new response.

Completion is a simulation done by fine-tuning on the behavior. The logprobs was twisted into not returning actual token numbers to make it less useful.

You can have it repeat back your last part (“start at the last line and continue composing”), and then continue writing, and then strip the matching start, if you want a technique that can slightly work sometimes.

1 Like