a) General chat based GPT like ChatGPT for users to ask and get answers on anything
b) Tech support and ‘how to’ answers for our platform
c) Named entity recognition for performing commands in our app
We are trying to figure out a system for which prompt to use when a user asks a respective question. As it stands, our system prompt just for tech support is well over 20,000 tokens (and growing, as we’re finding tons of errors and it turns out we need to give even more examples).
What we’ve tried so far: building a system prompt that classifies a user question as one of the 3 buckets, but it’s not performing well (we asked GPT to generate and optimize, and it works 50% of the time only). We’ve tried multiple ways. Also there’s a concern that we’re frivolously dishing system prompt for every query even when it is completely irrelevant. Example: ‘Tell me about Quantum Physics’ would have nothing to do with the 20K+ tokens for our system prompt that talks about our tech support and how to use the platform.
Ideas on what can we do that has worked for others?
A better approach, for your specific scenario, would be to use a single prompt (a) with tool support that does (b) and (c). So you could have a tool called help that’s used to answer ‘how to’ questions.
For your help desk prompt you should really be using a vector database and semantic search. This will eliminate the need to send 20k tokens for every help request.
Agree with @stevenic here. Instead of throwing everything at the model, you can simply offer the user options to choose what they want to do right in the chat interface. Then simply load the system prompt for the choice made by the user.
Thanks for this! We’ve built the vector database - great suggestion. However we’re still curious what to do about a case where we have 3 question types ie:
General (like ChatGPT)
Help (this is for answers to our application, this is the vector database now)
Commands (return JSON)
The question now is, how do we route a user’s question to the right bucket? Our users cannot be expected to toggle which bucket their question intention belongs to ie 1, 2, or 3. Instead, we’d like Chat Completion to determine, based on their question, which bucket to route them to. And when it’s ambiguous - we want it to ask the user for clarification.
We’ve tried to build an intent classification prompt which returns the intent of the question, but it has mediocre accuracy and we’re trying to make it airtight. If the question is still ambiguous, return a clarifying/confirmation question.
Here’s what we’ve written in the prompt:
You are an intent classification model that can classify user queries into one of the following TYPE: HelpQuestion, GeneralQuestion, Command.
respond with JSON {“Type”: “”}.
A command starts with one of the following words or a synonym: spawn create make build grab bring scale grow shrink delete.
A Help Question would be one of the following or with similar intent:
Example. User asks: ‘I want to go to NY’
Expected response should be a command (takes user to NY)
Actual response: General question