Thanks for updating the code with prompt.
However, the example you provided doesn’t have “response_format”:{“type”: “json_object”} in payload.
This will output json, but output may contains other text before/after json string.
Whereas JSON mode ensures that the output only contains a valid json string.
1 Like
This worked with ’ gpt-4-1106-preview’, you must have JSON in the message sent.
{
"model": "gpt-4-1106-preview",
"response-format": {"type": "json_object"},
"messages": [
{"role": "system", "content": "Answer in json format"},
{"role": "user", "content": "Hello whats up?"}
],
"temperature": 0.0
}
I also get “Unrecognized request argument supplied: response-format” all the time
1 Like
When we get JSON output, I’m still seeing that the message response is still just text and the “json” shows \n (newlines). Is this expected output?
1 Like
I’m also trying gpt4-vision and json output without success.
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {api_key}"
}
payload = {
"model": "gpt-4-vision-preview",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": prompt
},
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}",
"detail": 'low',
}
}
]
}
],
"max_tokens": max_tokens,
#"response_format": response_format, # This is not currently working on gpt-4-vision-preview
"response_format": { "type": "json_object" }
}
response = requests.post("https://api.openai.com/v1/chat/completions", headers=headers, json=payload)
I get this response:
{'error': {'message': '1 validation error for Request\nbody -> response_format\n extra fields not permitted (type=value_error.extra)',
'type': 'invalid_request_error',
'param': None,
'code': None}}
2 Likes
Yea I can get it to work with gpt-3.5-turbo-1106 and gpt-4-1106-preview, but not gpt-4-vision-preview.
3 Likes
I wonder if on the 1106 preview it hasn’t rolled out to everyone yet. I posted my response object above. Once again this is what I was using last night for the vision, not the gpt 4-1106
1 Like
You mean the code where you neither specify the response format nor use “json” in your API language and it only produces json-like output because you explicitly prompted for a “list” that is not a list?
Yes, that will work on the vision model.
(to get an error)
1 Like
I think they may have some bug in the rate-limiting. I tried my custom code again, but it failed. But the example they put on the website works fine. I just modified this to save the JSON. Maybe they are doing phased rollouts again on this stuff?
import base64
import requests
import json
# OpenAI API Key
api_key = ""
# Function to encode the image
def encode_image(image_path):
with open(image_path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode('utf-8')
# Path to your image
image_path = "captured_image.jpg"
# Getting the base64 string
base64_image = encode_image(image_path)
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {api_key}"
}
payload = {
"model": "gpt-4-vision-preview",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "What’s in this image?"
},
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}"
}
}
]
}
],
"max_tokens": 300
}
response = requests.post("https://api.openai.com/v1/chat/completions", headers=headers, json=payload)
# Save response to JSON file
with open('response.json', 'w') as f:
json.dump(response.json(), f)
print('Response saved to response.json')
{“id”: “chatcmpl-8IInG4nz5qVqtDcI4gCurIQwQHeus”, “object”: “chat.completion”, “created”: 1699372838, “model”: “gpt-4-1106-vision-preview”, “usage”: {“prompt_tokens”: 1118, “completion_tokens”: 54, “total_tokens”: 1172}, “choices”: [{“message”: {“role”: “assistant”, “content”: “The image appears to be completely black. If there is supposed to be content, it’s not visible due to the darkness or it might be an error with the image itself. If you were expecting to see something specific, please check the file and try uploading it again.”}, “finish_details”: {“type”: “stop”, “stop”: “<|fim_suffix|>”}, “index”: 0}]}
type or paste code here
3 Likes
I think I figured out how to get JSON mode to work reliably. This should work for GPT4, GPT3.5, and if you’re using function calling or streaming. You have to explicitly define the structure of the desired JSON in the prompt. Bonus points if you also show GPT an example.
Here, I use JSON mode to get gpt-4-1106-preview to generate a JSON object that looks like this:
{
"thoughts": <the AI's thought process>,
"response": <the response to show to the user>
}
Here’s the end of my prompt
Return a JSON object with the following structure:
- thoughts: Your thought process behind the response
- response: The response to send to the user
Example:
{
"thoughts": "Zachary wants to learn something new. Let's begin by reviewing what we learned last time, and then we can move on to something new.",
"response": "Last time we met, we learned about the difference between the particles は and が in Japanese and polite past tense verb conjugation. Would you like to review these topics or learn something new?",
}
And then I call OpenAI (NodeJS):
const responseStream = await openai.beta.chat.completions.runFunctions({
model: "gpt-4-1106-preview",
messages,
stream: true,
response_format: {
type: "json_object",
}
});
And this is what I’d get back:
{
"thoughts": "💭Zachary seems to be progressing well in his Japanese and is about to start the A3 class. It's important to review what he learned last time, but also to introduce new material that connects to his interests and his ambition to move to Japan. Since his knowledge of practical usage and vocabulary is at 30%, we should work on expanding his vocabulary in areas of his interest, such as modern art or hip-hop dance. Additionally, with a 25% proficiency in verb conjugation, we can continue building on that. I should also consider incorporating something visual to make the learning process more engaging and help solidify his understanding.",
"response": "Hello Zachary! Last time, we delved into particles like は and が, as well as polite past tense verb conjugation. I think it's a great time to expand your vocabulary in areas you're passionate about, like modern art and music. How about we start with learning some key terminology in Japanese related to these topics? We can also practice some new verb conjugations in the context of your interests. What do you say?"
}
2 Likes
That doesn’t seem to work for “gpt-4-vision-preview” for me.
It returns:
{"error"=>{"message"=>"1 validation error for Request\nbody -> response_format\n extra fields not permitted (type=value_error.extra)", "type"=>"invalid_request_error", "param"=>nil, "code"=>nil}}
1 Like
-H “Content-Type: application/json”
-H “Authorization: Bearer $OPENAI_API_KEY”
-d ‘{
“instructions”: “You are a personal math tutor. When asked a question, write and run Python code to answer the question.”,
“name”: “Math Tutor”,
“tools”: [{“type”: “code_interpreter”}],
“model”: “gpt-4”
}’
{
“error”: “API key is not set in environment variables”
}
(venv) willw@Wills-MacBook-Pro AA_demo
do you guys know why it keep saying API key is not set in environment variables" while I do have api key? I try to use the assistant api
same error for me: extra fields not permitted.
I had json working fine before this release, and trying to convert to this is failing so far
@PaulBellow hi Paul, is json mode available on assistant api? Thanks man
I believe the assistant needs an additional header.
Calls to the Assistants API require that you pass a beta HTTP header. This is handled automatically if you’re using OpenAI’s official Python and Node.js SDKs.
OpenAI-Beta: assistants=v1
The documentation is here.
In case my solution is a bad one, remember to be kind. Maybe I am an undercover AI.
1 Like
The documentation has been updated to address this.
The issue is that if the model doesn’t know it’s supposed to be outputting JSON, it will output whitespace forever, hence the error is thrown if the word “json” doesn’t appear somewhere (which as you’ve discovered isn’t perfect, but hopefully provides a good guide on how to fix it)
JSON mode is supported for gpt-4-1106-preview but not for gpt-4-vision-preview, as of Nov 7th.
2 Likes
Thanks @owencmoore. When will JSON mode be support for gpt-4-vision-preview?
In the Assistants API, JSON mode is automatically turned on for function call argument generation (i.e. function call arguments will always be parseable JSON, though not guaranteed to match the provided schema).
As of Nov 2023, we do not support JSON mode for assistant messages in the Assistants API, outside of function calling.
Confirming that I see the same. Function calling works as expected in gpt-4-1106-preview but not gpt-4-vision-preview.