How to add a default reply if no response matches the user input

API
1

For any service, it is not a rare case that some users ask non-sense questions or send a message that is not even related to the business purpose. For that case I want to use a default reply for example to say “Sorry, this is not part of our services.” if the user input does not match any answer from the dataset.

So how to add a default reply to the model to avoid it from sending inappropriate answer that does not match the user prompt?

2

What existing approach are you taking @Rariny ? Embeddings or fine-tune?

3

I use fine-tune to achieve this. I collected all the users input and add completion to any specific prompt.

4

Embedding is a better way to achieve this functionality. But if you’re using fine-tune, consider adding prompt completion pairs for the ‘not’ cases. Basically, these should include things thst you don’t do and the completion would have the message "Sorry, this is not part of our services".

Kind of like how we train image classification model and include the ‘not’ cases.

1 Like
6

I might have an approach that might help solve this for you using text-embedding-ada-002. After reading a previous comment it seems that it might actually be useful.

Basically, you can use the text-embedding-ada-002 model to get embeddings for the input string against the training dataset prompts that you used and compute the similarity. If the similarity metric passes above a certain threshold, you can request your model to generate a response, else simply return a generic statement. Here is the code that can be used for this approach.

import json
import numpy as np

def get_similarity(user_prompt):
  # Initialize the string which will contain the strings for similarity test. Includes prompt.
  list_strings = [user_prompt]
  with open("<Your JSONL file used for training>.jsonl", "r") as f:
    for line in f:
      data = json.loads(line)
      list_strings.append(data["prompt"]) # assuming prompt-completion pairs. To pick only prompts.
  resp = openai.Embedding.create(
  input=list_strings,
  engine="text-embedding-ada-002")
  similarity_scores = []
  embedding_a = resp['data'][0]['embedding'] # embedding for the user's prompt
  for i in range(1,len(list_strings)):
    embedding_b = resp['data'][i]['embedding']
    similarity_score = np.dot(embedding_a, embedding_b) # simple similarity metric
    similarity_scores.append(similarity_score) 
  threshold = 0.87
  if(max(similarity_scores) >= threshold):
    return gpt_response(user_prompt)['choices'][0]['text'] #get response from your model
  else:
    return "Some Generic Output"

A couple of things to note:

  1. This uses the most basic ways to get the similarities, there are other ways to calculate similarities from embeddings so you might find one of the other methods more suitable. For simplicity, I’ve used the simple dot product.
  2. This clearly increases the cost incurred as each input is going to use some OpenAI resources for calculating the embeddings. So if the prompt is very long or if you have a large amount of training data, cost is definitely going to increase. Though it uses a better model which results in good results with lesser cost incurred.
  3. Make sure to mess around with the threshold value as per your wish and use-case. Also, if you’re using a different method for calculating similarity.

Hope this helps!

1 Like
7

This is really helpful, thank you so much for this.