Once the context free grammar is built, is it fast to generate the allowed next tokens based on what was generated so far?
Introducing Structured Outputs in the API
JSON Schemas supplied with Structured Outputs aren’t Zero Data Retention(opens in a new window) (ZDR) eligible.
Does this mean the CFG is kept in cache forever?