Introducing Structured Outputs

[markcwoodworth]

That’s not a useful reply. If the latency is 10 seconds to 1 minute, there isn’t really a way to design for that. What we need to understand is how the caching is performed, so we know how often the penalty is incurred. I’ve asked in several places for more detail, but so far no response:

Structured Output Latency - How is Caching done?

From the release (paraphrased): The first request will incur a latency penalty of 10 sec - 10 min, depending on schema complexity. Subsequent requests will not pay this penalty as the schema will be cached Is the caching done at the API Key + Schema level, such that every schema requested for a given API Key will be “permanently” saved and all future requests, regardless of how far apart they are in time, will benefit from this caching? This implies persisting cached schemas for each API ke…