How to know which stop sequence was detected in a completion?

API
,
1

I’m using the Completions API and have multiple stop sequences that are passed into the request. If the stop sequences are stripped from the completion response, is there a way to know which stop sequence was “triggered”?

For example, suppose that I send a completion request like this:

response = openai.Completion.create(
  model='davinci',
  prompt='What is your favorite 3-letter word?',
 `stop=['FOO', 'BAR']`,
)

Let’s say the completion (without the stop sequence) is something like this:

My favorite 3-letter word is FOO because I like the Foo Fighters. I saw them in concert last summer... <continue until max_tokens>

But another possible completion might be:

My favorite 3-letter word is BAR because I like to drink and socialize... <continue until max_tokens>

Because “FOO” and “BAR” are both stop sequences, the model completion would stop generating tokens once it hits either one, so it would look like this:

My favorite 3-letter word is

On the client side, I would not know if “FOO” or “BAR” was generated since it is stripped from the completion that is returned in the response.

Is there any way to determine which stop sequence was triggered? Or is there maybe an option to include the stop sequence in the completion?

2 Likes
2

Are you using it on the Playground or did you implement the API in your own project? If you’re sending custom prompts to the API you can print the finish_reason of the response into your app. I’ll copy the response object from OpenAI’s API reference. Finish_reason is inside the choices array:

{
“id”: “chatcmpl-123”,
“object”: “chat.completion”,
“created”: 1677652288,
“choices”: [{
“index”: 0,
“message”: {
“role”: “assistant”,
“content”: “\n\nHello there, how may I assist you today?”,
},
“finish_reason”: “stop”
}],
“usage”: {
“prompt_tokens”: 9,
“completion_tokens”: 12,
“total_tokens”: 21
}
}

3

Thanks - yes, I’m implementing the API using the Python SDK. I see the finish_reason in the response, but it only tells you the whether the completion was terminated to to a stop sequence (i.e. "stop" or because it hit the token limit ("length").

In my case, I have more than one stop sequence and want to know which stop sequence was triggered when the finish_reason is "stop".

1 Like
4

Just for trial what happens if you send text like this below.
My favorite 3-letter word is BARBAR because I like to drink and socialize

5

Ok, now i get it. So finish_reason is just saying that something out of the stop array which was send in the initial request was used to stop the creation/completion. I guess they modified the API so that it’s sending back “stop” as a string instead of an object. The object would probably contain the information you need. Did you try to manually change it? Otherwise you’re right, it’s not part of the documentation but it would actually be very useful if the response would send back that information

6

I am guessing that it’s also case sensitive. So BAR and bar are two different stop sequences.

7

@DavidOS366 yes the stop sequences are just exact string matches, so they’re case sensitive, and also your example of “BARBAR” would simply stop at the first “BAR”.

The real-world example of why I’m trying to do this is that I’ve fine-tuned a conversation model to use a format where the AI’s response either ends in a user prompt (e.g. “USER:”) if the conversation should continue or an end-of-conversation marker "e.g. “–END OF TRANSCRIPT–” if it’s done. If I included both as stop sequences, then the completions stop correctly but since they’re stripped out, I can’t tell which one was triggered.

So for now, I’m only including one of stop sequences in the API call and then relying on a hacky regex parser to look for the other sequence(s). It works, but is not ideal and would be much more efficient if the API could just tell me which sequence it triggered.

8

You could try for B.ARBAR as stop sequence. So in your output you might get B.AR at the end, since it will stop at BAR and not B.AR

1 Like
9

Hi. I was facing the same issue and I found out that if you set echo=True the API will send you back your prompt plus the response stop sequence included.

[edit] I believed this, but it was actually not true. I don’t see the reason why to exclude the stop sequence from the response. On the contrary: having it would be of great benefit. I suggest to include it as an additional element in the choice object (e.g. stop_sequence = null or str) to keep compatibility with current specification.