Sorry for the very general question but I’m just starting to work on it now. Is it possible to extend gpt knowledge with technical domain information?
Let me explain. I work in a very technical field and would like to get more in-depth answers from gpt, as if it were a real domain assistant. Unfortunately, what I get are superficial answers, and I guess that is completely normal. Is there any way to provide it with material, tell it what documents to read or what sites to visit to improve the quality of content?
If so, then I will ask you how to do it
Build a system that searches for fitting documentations and crawls and embedds them into an ada model. When that process has finished you can start searching on it.
Thank you so much @jochenschultz . Does not seem like an easy thing to do, but maybe I can get a budget from my company. I’ll update.
If you are interrested we can do that together in a video session. I could imagine a few more ways to solve that.
You could add some context to your prompts by getting the data from another system first e.g.
- a wordbubble/knowledge tree in a RDBMS
- a graph DB that stores relations between documents which is filled over time by crawlers
- a vector db
- maybe even live crawling and summarizing
- a combination of above
You can even build a database extension e.g. for postgresql that enables your database to understand SQL like this:
SELECT content from querypool WHERE prompt=‘…’
That would be like a stored procedure with access to different architectures (even a pool of local PDFs)…
… and of course utilizing other/own models. Although there is not much even slightly as good as the openai world
The most common way to do this is with a vector database like Milvus, but as @jochenschultz mentioned, relational and graph databases are starting to be used as well. One of the most common ways of injecting domain data into GPT-3.5/4 is with Llamaindex.
Some blogs that might be helpful for this: