I am making GPT-4 API calls in a loop over chunks of data, and using json mode to return the results in JSON format. However, out of 20 calls, only 17 had the final closing curly bracket. Three in a row were missing them, despite coming in under half of the max tokens that were set for the chunk ( 454 vs 1150, 764 vs 2150, and 797 vs 2100). My logs don’t show that any errors or exceptions were triggered by the API response.
Hi Ashlynn
Could you explain your use case a bit more?
In many cases it makes sense to break down getting data and getting json. In this case you’d use 3.5-turbo-1106 with json mode.
I am working with novels (my own for now, I’m an author). I have an NER step, and then I send each chapter to GPT-4 to get information about each chapter and location that the NER model (GPT 3.5 Turbo) found.
My schema is
{
Characters: {
character_name: {
Appearance: description,
Personality: description,
Relationships: description
}
},
Settings: {
location_name: {
Appearance: description,
Location relative to other places: description,
Familiarity to the narrator: description
}
}
}
So what you are saying is that I should tell GPT-4 just to write a paragraph, and then have GPT-3.5 sort it into JSON?
Yeah, you can ask gpt-4 to write a paragraph and ask gpt-3.5 to do analysis and give you a json with the data you need.
Well, that was a waste of time. GPT-3.5 just returns me the blank schema every time
I might be able to stop the time wasting. Read about models:
response_format
An object specifying the format that the model must output. Compatible with gpt-4-1106-preview and gpt-3.5-turbo-1106.
Setting to { "type": "json_object" } enables JSON mode, which guarantees the message the model generates is valid JSON.
JSON don’t appear just by magic. You still have to fully specify them, almost to the quality where using this mode is more problematic than helpful because of its failure mode.
This, BTW is a schema:
I do have that. And I got valid JSON.
Of the schema I provided (shown above). It responded with my schema verbatim without the information from the paragraphs generated by GPT-4. I basically spent 4 cents to make 20 copies of my schema 
I’ll offer you an alternative approach (mentioned here) you can explore.
disclaimer: I feel like jsonmode, functions, assistants, and all that stuff are gimmicks. but that’s just my current personal opinion
this works well with gpt-3.5-turbo-instruct:
Here’s a short story. Could you kindly create a description of the story as a JSON object?
-----------Once upon a time in a small village, there were ten unique individuals. There was Amelia, the talented artist, always seen with a sketchpad in her hand. Next was Benjamin, the curious scientist, constantly conducting experiments in his backyard. Clara, the kind-hearted nurse, spent her days tending to the ill and the injured. Darren, the mischievous trickster, always had a prank up his sleeve. Eliza, the aspiring chef, could be found in her kitchen, concocting delicious recipes. Fred, the wise old man, spent his time under his favorite tree, sharing stories with anyone who would listen. Grace, the passionate teacher, inspired her students with her energetic lessons. Harry, the adventurous explorer, was always venturing into the unknown, seeking new discoveries. Isabella, the gentle gardener, nurtured her magnificent botanical creations. Lastly, Jackson, the compassionate volunteer, dedicated his time to helping those less fortunate. These ten individuals, though different in their pursuits, shared a common love for their village and one another. Their unique talents and selflessness came together, creating a harmonious community that thrived, celebrated, and most importantly, lived happily ever after.
---------
schema:{ Characters: { character_name: { Appearance: description, Personality: description, Relationships: description }, Settings: { location_name: { Appearance: description, Location relative to other places: description, Familiarity to the narrator: description } } }
--------
JSON:{
generation
"Characters":{
"Amelia":{
"Appearance":"Always seen with a sketchpad in her hand.",
"Personality":"Talented artist",
"Relationships":"Loved her village and the other villagers"
},
"Benjamin":{
"Appearance":"Constantly conducting experiments in his backyard.",
"Personality":"Curious scientist",
"Relationships":"Respected by the other villagers for his knowledge and discoveries"
},
"Clara":{
"Appearance":"Spent her days tending to the ill and injured.",
"Personality":"Kind-hearted nurse",
"Relationships":"Cared for everyone in the village, especially the sick and injured"
},
"Darren":{
"Appearance":"Always had a prank up his sleeve.",
"Personality":"Mischievous trickster",
"Relationships":"Brought laughter and joy to the village with his tricks, loved by all"
},
"Eliza":{
"Appearance":"Could be found in her kitchen, concocting delicious recipes.",
"Personality":"Aspiring chef",
"Relationships":"Cooked for the whole village, always sharing her love for food"
},
"Fred":{
"Appearance":"Spent his time under his favorite tree, sharing stories with anyone who would listen.",
"Personality":"Wise old man",
"Relationships":"Mentored the younger villagers with his wisdom, loved by all"
},
"Grace":{
"Appearance":"Inspired her students with her energetic lessons.",
"Personality":"Passionate teacher",
"Relationships":"Known and loved by all the village children for her inspiring teaching methods"
},
"Harry":{
"Appearance":"Always venturing into the unknown, seeking new discoveries.",
"Personality":"Adventurous explorer",
"Relationships":"Admired by the other villagers for his bravery and determination"
},
"Isabella":{
"Appearance":"Nurtured her magnificent botanical creations.",
"Personality":"Gentle gardener",
"Relationships":"Loved by the villagers for her beautiful gardens and kind nature"
},
"Jackson":{
"Appearance":"Dedicated his time to helping those less fortunate.",
"Personality":"Compassionate volunteer",
"Relationships":"Respected and appreciated by the villagers for his selfless acts of kindness"
}
},
"Settings":{
"Small Village":{
"Appearance":"A quaint and peaceful village, surrounded by lush greenery and fields.",
"Location relative to other places":"Located in a secluded area, far from any bustling cities.",
"Familiarity to the narrator":"The villagers were like family to each other, living and working together harmoniously."
}
}
}
just prepend a { to the generation and you have a valid json. I haven’t had an issue with this so far if the schema’s fairly simple. I’ve been using this method since davinci-002 came out.
Sometimes the text is too difficult to understand and you may need preprocessing steps/chunking or use gpt-4