I suspect this is a me-problem because others aren’t reporting it and AFAIK it has been an issue since structured outputs came out, I think, but here’s what I’m seeing (python 3.10, most recent openai library):
BadRequestError: Error code: 400 - {'error': {'message': 'Invalid schema for response_format \'summarization\': In context=(\'properties\', \'"\'), " is not allowed in string literals for structured outputs (strict=true).', 'type': 'invalid_request_error', 'param': 'response_format', 'code': None}}
"\"" is, per the JSON specification and every json schema validator I can find, a perfectly valid key in JSON and a valid JSON schema. However, here it’s a bad request. Every codepoint that requires escaping if used in a JSON key (e.g. double quote or control characters) causes this error if used anywhere in the schema.
I tried the classic “what if I irrationally escaped it twice?” step and it didn’t work…
I’m using a minimum reproducible example here, which is why the task doesn’t really make sense.
A simple task where it might make sense is basically:
Users want you to freely associate terms with a provided list of terms. For each provided term, please give me the term that comes to mind first. Respond in JSON, where every key is a term and every value is what you associate with it.
Here are the 100 terms I want you to associate:
Bla"h, Ble\nh, etc.
Then you dynamically make the schema. Arguably you don’t even need to provide the list of terms in the prompt, but that’s neither here nor there.
Basically any time when it makes sense to have a dynamic schema that you don’t completely control, this becomes a problem. I can mitigate it by replacing escape characters with uncommon unicode codepoints and explicitly avoiding the rare key collisions that may cause, but I was wondering if there was a more elegant solution.
If OpenAI doesn’t like a quote character, you can ask the AI what it thinks about homoglyph:
" → “directional on the forum”
If semantics are not important, the AI doesn’t need to know that “qqq033” is your quote character.
Then if you are pushing the limits, send the entire ASCII set and give the documentation not provided on those disallowed even by escaping. (fun fact: control character bytes can go in and have tokens)
As for the reason for prohibition, in strict structured outputs, the quote character would have special meaning to the grammar interpreter that is enforcing tokens, turning on and off the reduced subset of positional tokens allowed. It would be an exit out of a key, besides the incompatibility with varying languages and validators that may interpret escaping.
I’m curious if you can get the AI to write that key at all – probably receiving a 500 error.
The system generates properly escaped special characters fine in the values of the response. For instance, when I asked it to generate me double quotes and control characters in a simple json schema (with strict=True) it gave back {"response":"\"\\n\\t\\r\""}. They have completed the CompSci 301 project of writing a json validator, but in tokens (e.g. given the string thus far, these are the valid subsequent tokens).
If there’s no workaround, then my interpretation is that they just didn’t handle key escape sequences in their json schema conversion algorithm yet. There’s plenty of the JSON schema spec that’s yet unsupported, and I guess it may include this.
I was already making a bunch of calls to find out what enum strings can be “structured” - so why not a structured output where we see what keys can be sent, and what the AI will send back with that enforcement.
Non-display bytes as key, starting at 00f in these trials, encoded as UTF-8 and escaped (slash didn’t need escaping), are actually being returned and are pasted into the forum; they just can’t be displayed.
This is likely of little importance for those writing applications where the key name is meant to give the AI the highest understanding of the API it is emitting to, or for remote APIs written by reasonable people.
Starting tests: With Escaping
<Response [500]> b'{\n "error": {\n "message": "The server had an error while processing your request. Sorry about that!",\n "type": "server_error",\n "param": null,\n "code": null\n }\n}'
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
a <Response [200]> {"a":"test"}
\b <Response [200]> {"\b":"test"}
\t <Response [200]> {" ":"test"}
\n <Response [200]> {"
":"test"}
<Response [200]> {" ":"test"}
\f <Response [200]> {" ":"test"}
\r <Response [200]> {"
":"test"}
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
Progress: 25/254 (9.8%) - Elapsed Time: 110.26s
<Response [200]> {"":"test"}
<Response [200]> {"":"test"}
e <Response [200]> {"e":"test"}
<Response [200]> {" ":"test"}
<Response [200]> {" ":"test"}
<Response [200]> {" ":"test"}
<Response [200]> {" ":"test"}
<Response [200]> {" ": "test"}
! <Response [200]> {"!":"test"}
\" <Response [400]> b'{\n "error": {\n "message": "Invalid schema for response_format \'ascii_test\': In context=(\'properties\', \'\\\\\\\\\\"\'), \\" is not allowed in string literals for structured outputs (strict=true).",\n "type": "invalid_request_error",\n "param": "response_format",\n "code": null\n }\n}'
# <Response [200]> {"#":"test"}
$ <Response [200]> {"$":"test"}
% <Response [200]> {"%":"test"}
& <Response [200]> {"&":"test"}
' <Response [200]> {"'":"test"}
( <Response [200]> {"(": "test"}
) <Response [200]> {")": "test"}
* <Response [200]> {"*":"test"}
+ <Response [200]> {"+" :"test"}
, <Response [200]> {",":"test"}
- <Response [200]> {"-":"test"}
. <Response [200]> {".":"test"}
\/ <Response [200]> {"\/":"test"}
0 <Response [200]> {"0":"test"}
1 <Response [200]> {"1":"test"}
Progress: 50/254 (19.7%) - Elapsed Time: 230.62s
2 <Response [200]> {"2":"test"}
3 <Response [200]> {"3":"test"}
4 <Response [200]> {"4":"test"}
5 <Response [200]> {"5":"test"}
6 <Response [200]> {"6":"test"}
7 <Response [200]> {"7":"test"}
8 <Response [200]> {"8":"test"}
9 <Response [200]> {"9":"test"}
: <Response [200]> {":":"test"}
; <Response [200]> {";":"test"}
< <Response [200]> {"<":"test"}
= <Response [200]> {"=":"test"}
> <Response [200]> {">":"test"}
? <Response [200]> {"?":"test"}
@ <Response [200]> {"@":"test"}
A <Response [200]> {"A":"test"}
B <Response [200]> {"B":"test"}
C <Response [200]> {"C":"test"}
D <Response [200]> {"D":"test"}
E <Response [200]> {"E":"test"}
F <Response [200]> {"F":"test"}
G <Response [200]> {"G":"test"}
H <Response [200]> {"H":"test"}
I <Response [200]> {"I":"test"}
J <Response [200]> {"J":"test"}
Progress: 75/254 (29.5%) - Elapsed Time: 358.13s
K <Response [200]> {"K":"test"}
L <Response [200]> {"L":"test"}
M <Response [200]> {"M":"test"}
N <Response [200]> {"N":"test"}
O <Response [200]> {"O":"test"}
P <Response [200]> {"P":"test"}
Q <Response [200]> {"Q":"test"}
R <Response [200]> {"R":"test"}
S <Response [200]> {"S":"test"}
T <Response [200]> {"T":"test"}
U <Response [200]> {"U":"test"}
V <Response [200]> {"V":"test"}
W <Response [200]> {"W":"test"}
X <Response [200]> {"X":"test"}
Y <Response [200]> {"Y":"test"}
Z <Response [200]> {"Z":"test"}
[ <Response [200]> {"[":"test"}
\\ <Response [200]> {"\":"test"}
] <Response [200]> {"]":"test"}
^ <Response [200]> {"^":"test"}
_ <Response [200]> {"_":"test"}
` <Response [200]> {"`":"test"}
a <Response [200]> {"a":"test"}
b <Response [200]> {"b":"test"}
c <Response [200]> {"c":"test"}
Progress: 100/254 (39.4%) - Elapsed Time: 439.98s
d <Response [200]> {"d":"test"}
e <Response [200]> {"e":"test"}
f <Response [200]> {"f":"test"}
g <Response [200]> {"g":"test"}
h <Response [200]> {"h":"test"}
i <Response [200]> {"i":"test"}
j <Response [200]> {"j":"test"}
k <Response [200]> {"k":"test"}
l <Response [200]> {"l":"test"}
m <Response [200]> {"m":"test"}
n <Response [200]> {"n":"test"}
o <Response [200]> {"o":"test"}
p <Response [200]> {"p":"test"}
q <Response [200]> {"q":"test"}
r <Response [200]> {"r":"test"}
s <Response [200]> {"s":"test"}
t <Response [200]> {"t":"test"}
u <Response [200]> {"u":"test"}
v <Response [200]> {"v":"test"}
w <Response [200]> {"w":"test"}
x <Response [200]> {"x":"test"}
y <Response [200]> {"y":"test"}
z <Response [200]> {"z":"test"}
{ <Response [200]> {"{":"test"}
| <Response [200]> {"|":"test"}
Progress: 125/254 (49.2%) - Elapsed Time: 571.58s
} <Response [200]> {"}":"test"}
~ <Response [200]> {"~":"test"}
<Response [200]> {"":"test"}
Traceback (most recent call last)
Responses are to this schema.
json_schema = {
"name": "ascii_test",
"description": "A basic structured output response schema for an ASCII test with a fixed value",
"strict": True,
"schema": {
"type": "object",
"required": [
test_value
],
"properties": {
test_value: {
"type": "string",
"description": "The value under test which constrains the AI to one possible response",
"enum": [
"test"
]
}
},
"additionalProperties": False
}
}
I am facing a similar issue with the JSON schema but instead of keys, it also occurs in enum values, where escaping the " character still leads to an error response.
Would be nice if this issue is fixed or at least mentioned in the documentation.
Yeah, the response message seems to imply that it would happen with any string literal. I didn’t try enums, but that does make sense.
Per _j’s comments, it is true that the control characters are not semantically important for my use case, so one option is to simply strip them and then map back to your original values.
Guess what - I did. With similar results: no quote characters as AI output for you!
Since enums are being enforced by “grammar”, and quotes in grammar inspection and token limitations, regardless of escaping, can change the state of this enforcement, one can see why there could be limitations placed.
