We are building a Financial assistant GPT that talks to Quickbooks and Stripe. we added Open API schema for the Quickbooks API’s in the Knowledge section, in our experiments, it turn out that GPT is not using the knowledge base instead it is constructing the queries from the trained data, which is causing issues with certain endpoints query parameter generation. Did anyone have built a GPT that uses a Knowledge base we are happy to bounce ideas with you. I appreciate any help or suggestions on this. Also, we are seeing a new behavior today with knowledge retrieval rather than using the vector search GPT is using a code interpreter to construct the endpoints. Did anyone notice this? (
3 Likes
Similar experiences with far simpler GPT and straightforward, limited knowledge base files not being reviewed. Updated Instructions and Prompts to instruct GPC to review the knowledge base files and that seems to be working better. But if we continue prompting GPC reverts to ignoring knowledge base after 5-7 follow up prompts that lack the reminder. Hoping to have better results with Assistant.
1 Like
Same issue here. I get better results if I expliciltly ask the assistant to retrieve information from its knowledge base, but it’s definitely not a robust solution.
For instance, when given up-to-date coding conventions, assistants still use outdated practices. When given documentation, assistant will still provide incorrect answers even though answers are explicit in the documentation.
1 Like
It uses the knowledge base information but it is just not intelligent enough. It often needs my intelligence to guide it. It diminishes the abilities of AI.
Not only that. The biggest problem is that it often does not construct the queries based on the OpenAPI spec. When it does, it sometimes does not choose the best endpoint for a prompt. How do I know that? I ask its rationale of choosing a particular endpoint. Eventually, it would admit it has not chosen the best endpoint. Again, it needs my intelligence to guide it to make the right choice. That defeats the very purpose of AI.
1 Like
It doesn’t seem to remember anything! I tried adding basic knowledge docs, priming initial instructions multiple different ways, reminding it periodically, etc… it never remembers… I guess I am better off using the main chatGpt.
I have seen some success with extending instructions for particular activities into my knowledge base. For example…
IMPORTANT! If a user asks you to create a recipe, consult your knowledge file recipe-instructions.md first!
This has been fairly good. Maybe it goes back to the “distance in prompt history” theory that many users have proposed?
I should mention though that it didn’t work at first, I had to add the “IMPORTANT!” part.
Now I’ve started playing with Actions and made a similar thing for how to handle API calls. This has not been nearly as successful though. It jumps the gun and makes the calls despite my knowledge-based instructions telling it to collect more information first. Hallucinating API Actions' required params