I’m seeing a strange behavior with both GPT-4 and 3.5 when output is a list of items. Lists never get formatted well (see below).
Here’s the input I’m using:
[{'role': 'system', 'content': 'Help the user with his query'}, {'role': 'assistant', 'content': 'Hi, welcome to the chat! How may I help you?'}, {'role': 'user', 'content': 'give me a list of 10 male names'}]
And here’s the response:
{
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "Sure, here's a list of 10 male names:\n\n1. James\n2. John\n3. Michael\n4. William\n5. David \n6. Robert \n7. Joseph \n8. Thomas \n9.Charles \n10.Richard ",
"role": "assistant
}
}
],
"created": 1697205875,
"id": "chatcmpl-89D4FoEwdleiNLczjuOjjJQk9m92n",
"model": "gpt-4-0613",
"object": "chat.completion",
"usage": {
"completion_tokens": 53,
"prompt_tokens": 213,
"total_tokens": 266
}
You can see that list gets messed up with some blank spaces around the items. This never happens in the Playground, everything is formatted well:
You just need to increase the certainty of what it is going to generate with language, and decrease the allowed uncertainty with API parameters.
import os
import openai
openai.api_key = os.getenv("OPENAI_API_KEY")
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content":
"give me a list of 10 male American English names, "
"output in markdown format."}]
completion = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
top_p=0.2,
messages=messages)
print(completion.choices[0].message.content)
Sure! Here is a list of 10 male American English names in markdown format:
On the other hand, I don’t need to provide these instructions or lower the temperature in Playground (or Chat) to make it work. Why would I need to do it via API?
The real question is - why is this not working with API, when it works through Playground or regular Chat via https://chat.openai.com
Well, now that you can see the problem solved but don’t want to use the techniques to solve it - at least you can explore assigning a better identity to the AI in the system role to let AI be a chatbot that will follow conversations and prefer specific output styles. An assistant output first may degrade some of the AI fine-tuning to chat successfully.
You will get identical results if you provide identical parameters. You have repetition penalties cranked up in the playground screenshot, which is not necessary for chat models without specific purpose.
Press “view code” and see that you are using playground parameters and inputs exactly.
I appreciate your help and the sarcasm
It seems like lowering down frequency_penalty and presence_penalty did it. Not sure why tbh since I left it at 0.85 in the Playground and it worked without issues. It didn’t have any relation to the prompt though, I’ve left it as is. Same goes for top_p and temp.
