Is there a way to force function to return a minified json?

API
,
1

I am trying to get output in JSON using functions, using GPT 3.5
I have defined a multilevel function.

The JSON output returned by the functions have unnecessary whitespaces, which I assume is to indent the json. Now these white spaces are counted as a token. Creating unnecessary bill and reducing available context size.

Is there a way to force function to return a minified json?

The output from API. Here in the arguments you will see a lot of whitespaces

[
    {
        "index": 0,
        "message": {
            "role": "assistant",
            "content": null,
            "function_call": {
                "name": "print_grammatical_error",
                "arguments": "{\n  \"gramatical_errors\": [\n    {\n      \"errorLocation\": {\n        \"start\": 0,\n        \"end\": 7\n      },\n      \"mistakeText\": \"Although some\",\n      \"mistakeReason\": \"Missing comma after introductory phrase\",\n      \"mistakeCorrection\": \"Although, some\"\n    },\n    {\n      \"errorLocation\": {\n        \"start\": 9,\n        \"end\": 17\n      },\n      \"mistakeText\": \"indivuduals\",\n      \"mistakeReason\": \"Spelling mistake\",\n      \"mistakeCorrection\": \"individuals\"\n    },\n    {\n      \"errorLocation\": {\n        \"start\": 19,\n        \"end\": 27\n      },\n      \"mistakeText\": \"shoud\",\n      \"mistakeReason\": \"Spelling mistake\",\n      \"mistakeCorrection\": \"should\"\n    },\n    {\n      \"errorLocation\": {\n        \"start\": 29,\n        \"end\": 38\n      },\n      \"mistakeText\": \"recieve\",\n      \"mistakeReason\": \"Spelling mistake\",\n      \"mistakeCorrection\": \"receive\"\n    },\n    {\n      \"errorLocation\": {\n        \"start\": 40,\n        \"end\": 46\n      },\n      \"mistakeText\": \"\\\"abuse anymore\\\"\",\n      \"mistakeReason\": \"\\\"Anymore\\\" should be placed before \\\"abuse\\\"\",\n      \"mistakeCorrection\": \"\\\"anymore abuse\\\"\"\n    },\n    {\n      \"errorLocation\": {\n        \"start\": 56,\n        \"end\": 62\n      },\n      \"mistakeText\": \"\\\"agreeded\\\"\",\n      \"mistakeReason\": \"\\\"Agreeded\\\" should be \\\"agreed\\\"\",\n      \"mistakeCorrection\": \"\\\"agreed\\\"\"\n    },\n    {\n      \"errorLocation\": {\n        \"start\": 64,\n        \"end\": 72\n      },\n      \"mistakeText\": \"recente\",\n      \"mistakeReason\": \"Spelling mistake\",\n      \"mistakeCorrection\": \"recent\"\n    },\n    {\n      \"errorLocation\": {\n        \"start\": 74,\n        \"end\": 82\n      },\n      \"mistakeText\": \"\\\"similars emotions\\\"\",\n      \"mistakeReason\": \"\\\"Similars\\\" should be \\\"similar\\\"\",\n      \"mistakeCorrection\": \"\\\"similar emotions\\\"\"\n    },\n    {\n      \"errorLocation\": {\n        \"start\": 84,\n        \"end\": 91\n      },\n      \"mistakeText\": \"\\\"anymore to use animals\\\"\",\n      \"mistakeReason\": \"\\\"Anymore\\\" should be placed before \\\"to use animals\\\"\",\n      \"mistakeCorrection\": \"\\\"to use animals anymore\\\"\"\n    },\n    {\n      \"errorLocation\": {\n        \"start\": 128,\n        \"end\": 137\n      },\n      \"mistakeText\": \"\\\"recognize worldwide\\\"\",\n      \"mistakeReason\": \"\\\"Recognize\\\" should be \\\"recognized\\\"\",\n      \"mistakeCorrection\": \"\\\"recognized worldwide\\\"\"\n    }\n  ]\n}"
            }
        },
        "finish_reason": "stop"
    }
]```
2

You can do this with your own code before sending the text again.

3

I do not intend to make a new API call with this JSON in the prompt again. My usecase require to get this JSON… no more calls using this as the context.

My main concern is, this extra unnecessary space are counted as tokens, hence billed. I guess doing anything minifying in my code would not solve the issue …

4

Your api could minify it before it answers to the function call. So you add it where the api outputs the json.

5

I don’t think that’s what the OP is asking. Essentially you have to pay almost 2x!
The money you need to pay depends on how many tokens OpenAI sends you (output tokens dominate input tokens in both price and latency).

Sure, you can reduce num_input_tokens as you suggested for the next call, but the fact that you already got billed for what OpenAI sent you is out of your hands (unless they provide the option). In many cases there might not even be a next round or you might even be able to omit parts of the function call to reduce tokens.

1 Like
6

The tokenization means that long series of characters such as indentation are only one token. One space is the same as eight spaces, two backslashes = 1. However, it’s trying to make the arguments pretty, and it’s been “trained” by the way the function is written and presented to it.

Your “arguments” after being unescaped:


"{
  "gramatical_errors": [
    {
      "errorLocation": {
        "start": 0,
        "end": 7
      },
      "mistakeText": "Although some",
      "mistakeReason": "Missing comma after introductory phrase",
      "mistakeCorrection": "Although, some"
    },
    {
      "errorLocation": {
        "start": 9,
        "end": 17
      },
      "mistakeText": "indivuduals",
      "mistakeReason": "Spelling mistake",
      "mistakeCorrection": "individuals"
    },...

You can try to guide it with a prompt example when you actually have more to be saved by a large output than the instruction itself, say documents using above 4k of context. Of course we assume you’ve already tried some prompting: “avoid whitespace and carriage return”, “use unescaped single quotes in JSON function arguments” “concatenate function argument to single line”, etc.

Also, you can look at tiktokenizer and your function and see that some of the parameter terms themselves could be rewritten to save you a few tokens for every instance of a correction, for example “mistake”-> “error”. Spelling “grammatical” correctly. Using underscores instead of camelCase. etc.

2 Likes
7

The model seemed not to know what is “minified”, so most of the times it just spits random formats, sometimes with indents, sometimes don’t.