I am spiking on creating LLM based applications around a framework I am working on. I would like to generate a JSON document based on a grammar. The grammar hasn’t been formalised in any notation yet but there’s a JSON schema and a public facing documentation available.
The main obstacle I am facing is that the current models do not possess any understanding of the JSON grammar, even though there’s a publicly available documentation of it (it seems that neither GPT-3.5 nor GPT-4 have been trained on those data).
From my understanding there are several approaches to augment the context of LLMs:
- Augmenting prompts with in-context learning
- Using embeddings and information retrieval
- Fine-tune a base model
- Train your custom model
Note: If you know other alternatives feel free to comment on this post, I would be very curious to learn about alternatives.
In the grammar, there are quite some permutations possible so I am wondering if few-shot prompting is a good solution for such use case. I am afraid that injecting the grammar context in the prompt is going to end up clearing up the context window fairly quickly (even though it seems that this might not be a problem in a near future).
Just curious, what would be the most adequate approach in term of:
- quality of the result
- API processing / response time
- cost effectiveness
At this point, the focus is about getting reliable results that are qualitative.
Note, i’ve tried function callings but it seem that they are designed to cover use cases when your function doesn’t take a fairly complex object. At least, it didn’t work against the current JSON schema that we have erroring out on some enums. It’s also unclear which Draft version of the JSON schema they do support.