Classify whether a question can be answered from the provided data

[curt.kennedy]

You can certainly do it without embeddings using GPT-3 with a fine-tuned Ada or Babbage. More details below, but I do this regularly, and it is one of the hidden gems of GPT-3 as compared to other solutions out there. It’s quick, accurate, and relatively inexpensive too to run these fine-tuned models.

Do you fine tune? If so why?

Just curious… Do you fine tune your model and if so why? Are you trying to reduce the number of tokens you pass in? If so what’s the cost trade off? Fine tuned models are expensive… At Microsoft, lots of people are saying to avoid fine tuned models. They’re expensive and its not clear there’s a benefit over a well written prompt. Just curious what others in the community are finding…

How to "escape" user input?

You could also train a classifier with one token output (’ 1’, ’ 2’, …, ’ 10’). Then force the temperature to 0 and set it to max 1 tokens. That will fence it in so much, and there is no such thing as a prompt in the classifier, so no worries about a prompt injection. Plus trained classifiers perform well with lower (cheaper/less $$$) models such as Ada or Babbage. You can even use GPT-3 to create the training dataset for the classifier if you feel it’s accurate enough in its raw capabilities…