Ranking prompts by relevance

Prompting
1

As title suggests, I want to rank prompts by relevance. Suppose you have a list of reviews from customers on a website and you want to find which review is the most helpful one, how should I approach that? Thank you in advance.

1 Like
3

If you’re able to clearly define what “helpful” means, you can “spell-it-out” to the model to improve results.

For example, I would include a description of the product in question, the customer review and then train the model to output individual scores for a set of properties.

Product: Harry Potter full series box set.
Review: Both my kids loved it!
Rank:
Relates to product: 0
Satisfaction: 10
Usefulness: 7
Dangerous: 0
Fragile: 0
Meet Expectations: 8
Overall Helpfulness Score: 25

###

In a training file this example will look like this:

{"prompt": "Product: Harry Potter full series box set.\nReview: Both my kids loved it!\nRank:\n", "completion":" Relates to product: 0\nSatisfaction: 10\nUsefulness: 7\nDangerous: 0\nFragile: 0\nMeet Expectations: 8\nOverall Helpfulness Score: 25\n###\n"}

…where every prompt ends with “\nReview:” to indicate the end of the review text and every completion ends with “\n###\n” as a stop sequence for generation.

Instead of numbers, you could use labels such as low, medium, high. I find the models respond better to words rather than numbers, but perhaps it’s just my imagination.

You’ll only be using the last number from the output, can easily extract that with a regex.

I’ve used such step-wise approaches with good success, I personally like them. You can read about this approach in the documentation, https://beta.openai.com/docs/engines/focusing-on-one-task-at-a-time

1 Like
4

Hi. Sorry for the delayed response. I’m thinking along the lines of:

Prompt:
Reviewer 1: This phone has an awful camera, but the display is very slick
Reviewer 2: Bad phone, don't buy.

Output: [Reviewer 1, Reviewer 2]

Such that it points out which review is the most helpful one

1 Like
6

This isn’t exactly what I’m looking for, but focusing on one task at a time is very helpful. Thanks!

2 Likes