How do function schemas impact token limits/costs?

The model has to be presented with at least a sketch of the function definition passed in with the call and that can’t be free (cost or token length wise) so how does that impact things? I don’t know that I saw any details on this in the announcement…

I’ll just answer my own question on here since OpenAI is unlikely to ever answer this question themselves :frowning:

The answer is “most likely” that every token of function schema description you send counts as tokens on the input side of things. They’re likely appending a bit of instruction plus schema to your prompt which you’re paying for. They’re also likely passing in the entire JSON schema as part of the prompt, which I will tell you is a complete waste of energy as the model will ignore half of it. In Alpha Wave, I send a sketch of the expected schema to the model which is a 3rd of the size token wise and works just as well. A full schema is needed if you want to actually validate the response (they don’t appear to be doing that) but the model doesn’t need to see the full schema.

I’m 50% through implementing function support for Alpha Wave and I’m planning to support their new function_call feature as they’ve designed it but I’m going to give you a switch to disable using their function support in favor of just using Alpha Wave. This will likely save you money and work better. Not trying to be snarky I just hope to help educate the community as to why OpenAI can’t properly solve this problem on their end. It takes multiple model calls to solve it and they’d have to bill you for that which isn’t easy for them to do in a way that’s transparent. Not their fault… They just can’t properly solve this…

1 Like