A simple solution is the include a description on what kind of answer you are looking for in the prompt to lead GPT towards the answers you are looking for. But if you already try that and if it doesn’t work, you can try more expensive approach.
I have created a spreadsheet you can use to test it out this process. See How to use custom data with GPT3 without fine tuning
1. Create two categories of phrases, one for approval and one for disapproval. In this example Row 2 is consider as approval and Row 3 is disapproval, you can add more text if you need.
2. Use Cosine Similarly to validate the GPT response with your embeddings, notice on row 3 we have detected that the reply is not what you are looking for because it belongs in the disapproval category.
3. So we can ask GPT to paraphrase it to something better
Technical Summary
We are using a text classification approach to determine what type of results we are getting, and using a text generation approach (GPT) to get the results until we are satisfy.
Obviously this approach will have a higher cost so you might want to put a max number of retry to limit your API usage.
Let me know if you have any other questions, I’ll be glad to help.