I think we are on the same page. 
Signed up for the wait list already as I would like to integerate Prolog into the mix, e.g
or other more complex task
Constraints
Planning problems
Parsing with DCGs
Having a plugin that can talk with a Prolog back end would definetly be a game changer for such problems. I don’t know how much this dove tails in with what OpenAI seeks as noted with evals
We are looking for evals in the following categories:
- Over-refusals
- Safety
- System message steerability
- In-the-wild hallucinations
- Math / logical / physical reasoning
- Real-world use case (please describe in your PR how this capability would be used in a product)
- Other foundational capability
but with a Prolog plugin don’t really see a need for creating evals for Prolog as OpenAI can just siphon off of the connection for what they want.