I have created an RAG application using the GPT-4 API. The database consists of state-wide real estate regulations and related publications, and the anticipated user base will be real estate students, agents, brokers, firms, publications and lawyers. These will be busy, probably low (relatively speaking) computer skills people looking to quickly get a good answer to issues they are dealing with. What I mean to say is that most of these people will probably NOT have used ChatGPT or similar tools before.
That said, how do you quickly teach people like this the prompting skills they need to get the best answers possible? I think it’s safe to say that we all agree there really isn’t a one-prompt-fits-all model, even when you have a dataset that is focused in one area of expertise.
Surprisingly, the app does very well (when using GPT-4) at answering most questions. So far, most of the real estate exam questions I have fed it (when I was able to get it to understand the question) have been answered correctly.
And, there is the rub, because these exams are designed for humans capable of critical thinking and with a basic knowledge of the world in general, and real estate law in particular, that the AI doesn’t possess. It can only regurgitate what it is fed.
So, the person asking the question really needs to be fully aware of this, that the manner in which the question is formed is almost as important as the question itself.
In the documentation I have created so far, I try to stress these points:
- be as specific and clear as you can
- do not assume the AI knows anything, because it doesn’t
- the AI doesn’t think, it reacts
- do not depend on keywords because the AI evaluates based upon context (meaning, ideas), not keywords
- if you don’t get an answer the first time around, try re-phrasing, re-wording or even re-thinking your question
Any other suggestions? There has got to be more than a few of you out there with similar query systems providing technical documentation in various professional areas. I’m sure you are running up against the problem of very simple questions not being answered correctly, not because the information was not found, but because the AI didn’t understand the question.
How are you dealing with this?
Any and all responses appreciated.