Need concept validation Search to Completion

Hi guys,

This is my first “working” question. I needs some help to validate my approach to a tool that will generate replies based on prompt and previous replies validated by human. I’m staying schematic, sorry for that.

Given the fact that completion can not have files uploaded to serve as examples (is this the case?) and my project will definitely need several rather long examples in prompt (limited to 2K+ tokens), I was thinking to make the structure like following:

1 Create and upload jsonl file with a lot of examples and meta data
2 Take user input and perform a semantic search in the file to find several inputs similar to user’s
3 Take found documents and use them along with user input to construct the final prompt that will contain the examples but be below size limit
4 Run completion with the prompt to get generated reply
5 Run the reply through the filter to see if it is safe to return (go to 3 if not for max 3 times, then go to 2 asking to modify input)
6 Show generated text to user and ask if they keep it or would like another variant (go to 4 in this case)
7 Ask if user would like to take the result as is (go to 9) or modify it
8 Let user modify the text
9 Ask if user wants to complete the current text (attach it to the prompt and go to 4 with it) or they are done
10 User is done - take the input, combine with the final result, meta data and append to local examples file
11 Give the user final result
12 Replace the example files of the model with the new local file

Let me know if I miss something in the concept here. Thanks a lot.

1 Like

That’s correct.

The Answers endpoint is a one step process. Simply send an input to the endpoint with an uploaded file (one step, not three).

If you’re using the Answers endpoint, and you’ve manually created or verified the documents for being safe, then you likely wouldn’t need the content filter (after all, you’ll know the possible space of completions). If desired, yes, that’s one way of going about using the content filter.

Re-running the Answers endpoint with the same query is unlikely to have a different completion, so going back to step 4 may not give the user a new output.

If you’re combining the final result and appending user input, you’re largely left with the same documents, except that you’re adding in potentially incorrect user info, so I’m not sure how useful this would be.

@joey Thanks for the feedback. I think I was not clear about the purpose of the thing. I need the AI to reply to reviews and not answer questions about the reviews present in the dataset. Unless I’m wrong about the purpose of the Answers endpoint :

Given a question, a set of documents, and some examples, the API generates an answer to the question based on the information in the set of documents. This is useful for question-answering applications on sources of truth, like company documentation or a knowledge base.

What I need the AI to do is something like this:

Review:

Our first time to stay at Dauphin Island – we usually stay in North Florida or Orange Beach. The kids loved picking up the seashells, the fishing for spec’s from the surf was very good, the house was well laid out for 8 to 10 people, the sand was white and nice and clean, the response times to get the air conditioner repaired and the bathtub drain to be cleared was really good – overall a great place to stay. Also, the bakery on the island was super nice to go to each morning for breakfast.

Reply:

Thank you so much for staying with us and taking the time to write this wonderful review. It means a lot that you and your family had such a great time at Beach Break. I am sorry to hear about the AC repair and drain issue, but I am glad that those were resolved quickly. Thank you for giving us another chance, we look forward to seeing you and your family again soon!

That’s a real life example with the reply generated by davinci-instruct engine. Current issue is that the model pays too much attention to issues mentioned in the review even if the review is overall extremely positive, thus leaving the question for reply reader about the issues in the property instead of highlighting the key features. Drafting the prompt is not enough to get the “almost perfect” result in this case, that’s the reason why I need to have a “dynamic” prompt based on replies already created by humans (When the prompt was given examples of replies to similar reviews, it was producing much better output from the marketing point of view).

The goal to append the user feedback to the prompt is to allow the user to direct the reply towards the desired result or simply complete the reply being uncomplete.

Example of direction:

Reply by AI:

Thank you so much for staying with us and taking the time to write this wonderful review. It means a lot that you and your family had such a great time at Beach Break. I am sorry to hear about the AC repair and drain issue, but I am glad that those were resolved quickly. Thank you for giving us another chance, we look forward to seeing you and your family again soon!

Human edit:

Thank you so much for staying with us and taking the time to write this wonderful review. It means a lot that you and your family had such a great time at Beach Break. I am happy our maintenance team was fast to respond, we always pay

AI completion of human edit:

Thank you so much for staying with us and taking the time to write this wonderful review. It means a lot that you and your family had such a great time at Beach Break. I am happy our maintenance team was fast to respond, we always pay particular attention to keeping our properties clean and well maintained. The bakery sounds wonderful and I am glad you enjoyed it. Thanks again for taking the time to review us so we can continue providing the best vacation rentals.

Example of reply uncomplete:

Reply:

Thank you so much for staying with us and taking the time to write this wonderful review. It means a lot that you and your family had such a great time at Beach Break. I am happy our maintenance team was fast to respond, we always pay particular attention to keeping our properties clean and well maintained. The bakery sounds wonderful and I am glad you enjoyed it. Thanks again for taking the time to review us so we can continue providing the best vacation rentals.

Complete reply:

Thank you so much for staying with us and taking the time to write this wonderful review. It means a lot that you and your family had such a great time at Beach Break. I am happy our maintenance team was fast to respond, we always pay particular attention to keeping our properties clean and well maintained. The bakery sounds wonderful and I am glad you enjoyed it. Thanks again for taking the time to review us so we can continue providing the best vacation rentals. Kind Regards, Adam.

Again this is a working example.

I see, thanks for the clarification!

To get a more optimistic bot, I’d try providing instructions for the bot to be strictly positive and optimistic, as well as examples where the bot is strictly positive.

In general, to achieve results more like the examples you’ve provided, you’ll simply want to provide those examples to the API.

1 Like

Yes, we are working on the prompt testing the variations to get the result we want.

Exactly, that’s the reason I need the semantic search functionality to find the best match examples from the dataset to include in the prompt instructions as examples to improve the reply.

Seeing the above, what are your thoughts on the approach? Any feedback would be welcome. Thanks.

I think it does so because it was trained on too large web data (written by average manager’s who often pay too much attention to those details) without being clearly instructed about the TODOs and DONT DOs. That’s our goal - teach the model to provide a reply that helps business more than what humans produce in general.

I see, thanks for the clarification! I think it’s a great approach.

1 Like

Summary created by AI.

Sergeliatko wanted to validate his concept for creating a tool that would generate replies to reviews using AI and discussed his proposed structure. He pointed out that the AI seems to focus too much on the negative aspects of reviews, even when the reviews are overall positive. His goal is to have a “dynamic” prompt based on replies created by humans. Sergeliatko also provided a specific example of a review and the desired type of ideal reply.

In response, Joey affirmed a few parts of Sergeliatko’s process, including that completions cannot have files uploaded as examples and suggested that the Answers endpoint would simplify several steps. Joey pointed out that re-running the Answers endpoint with the same query is unlikely to provide a different output and expressed doubt about the usefulness of combining the final result with user input.

Addressing Joey’s feedback, Sergeliatko clarified that his tool is specifically for responding to reviews, and not just answering questions about them. He explained the goal of appending user feedback to the prompt and gave examples of how an AI reply might be edited and completed by humans. Sergeliatko agreed with Joey’s advice about making the bot more strictly positive and optimistic.

In closing, Joey acknowledged Sergeliatko’s clarification and praised his approach as a great one. ref

Summarized with AI on Jul 22
AI used: gpt-4-32k