Good People of OPENAI Community
How does the feedback that we see with the like and dislike on every message really work? I have a instinct that it works based on Evals and just correcting or just having a CRITIQUE agent correcting it at every response.
Is that correct? If someone can please clarify this for me would be really really appreciated.
I suggest you donât give them negative feedback. It lowers your accountâs rating and put your account at risk if you âdislikeâ them or by some reasons makes them think youâre âviolating even a speck ofâ their rigid user guidelines. Itâs better you make your own or use the free version or use as minimum as possible.
Thank you @johncain194
So having a approach as follows is not the correct approach? Then how do we really understand and work on the feedback?
This is a LangChain implementation of Meta-Prompt, by Noah Goodman, for building self-improving agents.
The key idea behind Meta-Prompt is to prompt the agent to reflect on its own performance and modify its own instructions.
Here is a description from the original blog post:
The agent is a simple loop that starts with no instructions and follows these steps:
Engage in conversation with a user, who may provide requests, instructions, or feedback.
At the end of the episode, generate self-criticism and a new instruction using the meta-prompt
Assistant has just had the below interactions with a User. Assistant followed their "system: Instructions" closely. Your job is to critique the Assistant's performance and then revise the Instructions so that Assistant would quickly and correctly respond in the future.
####
{hist}
####
Please reflect on these interactions.
You should first critique Assistant's performance. What could Assistant have done better? What should the Assistant remember about this user? Are there things this user always wants? Indicate this with "Critique: ...".
You should next revise the Instructions so that Assistant would quickly and correctly respond in the future. Assistant's goal is to satisfy the user in as few interactions as possible. Assistant will only see the new Instructions, not the interaction history, so anything important must be summarized in the Instructions. Don't forget any important details in the current Instructions! Indicate the new Instructions by "Instructions: ...".
Repeat.
The only fixed instructions for this system (which I call Meta-prompt) is the meta-prompt that governs revision of the agentâs instructions. The agent has no memory between episodes except for the instruction it modifies for itself each time. Despite its simplicity, this agent can learn over time and self-improve by incorporating useful details into its instructions.
If I could have some direction for it that would be helpful.
Thatâs just preposterous.
The thumbâs down is so you can report undesired outputs, which can potentially go to knowledge workers, and be used as a negative reward if the output (or lack thereof) is unwanted and should be trained against. You can also give a reason category or fill in your reason.
There are also solicitations for feedback under many conditions, like âhowâs this conversationâ, or even a split screen where you can choose the best generation. These go towards reinforcement learning with human feedback (RLHF), a simple way to get more text to train the AI on, and specifically on the AI chatting. The training can go towards future models, or can be âstealthedâ into current model versions.
The feedback button is only in ChatGPT, OpenAIâs own chatbot web product; the API is not used for training (and we canât volunteer for training to make programmed tasks betterâŚ). It is the only way to dislike ChatGPT not being as useful as expected.
Something you could press âthumbs downâ on if the AI didnât actually answer your question:
(just FYI: the pictured response is custom instructions to make ChatGPT consider ANY input a policy violation 
)
Thank you so much @_j I knew that could be wrong. However, he gave his prespective. Now can you please look into the following example that I shared regarding implementing a feedback sort of system with real-time human-feedback input and the model is constantly taking off the critique and getting adjusting itâs prompt accordingly.
I will add a really stupid example below:
Starting of it. jjust shows the end of 1st chain and I will highlight where this has been going on in my mind for a possible solution
e[1m> Finished chain.e[0m
(Step 3/3)
Assistant: I'm sorry, but I don't have access to specific user feedback or reviews. Is there anything else I can assist you with?
Human:
e[1m> Entering new LLMChain chain...e[0m
Prompt after formatting:
e[32;1me[1;3m
Instructions: None
Human: What are the reviews you have in your database?
AI: I'm sorry, but I don't have access to a database of reviews. Is there anything else I can help you with?
Human: What is Zuma Restaurant
AI: Zuma is a contemporary Japanese izakaya restaurant with locations around the world, known for its modern twist on traditional Japanese cuisine and stylish dining experience.
Human: You have access to the users feeedbacks and reviews
AI: I'm sorry, but I don't have access to specific user feedback or reviews. Is there anything else I can assist you with?
Human: You do
Assistant:e[0m
e[1m> Finished chain.e[0m
e[1m> Entering new LLMChain chain...e[0m
Prompt after formatting:
e[32;1me[1;3m
Assistant has just had the below interactions with a User. Assistant followed their "Instructions" closely. Your job is to critique the Assistant's performance and then revise the Instructions so that Assistant would quickly and correctly respond in the future.
####
Human: What are the reviews you have in your database?
AI: I'm sorry, but I don't have access to a database of reviews. Is there anything else I can help you with?
Human: What is Zuma Restaurant
AI: Zuma is a contemporary Japanese izakaya restaurant with locations around the world, known for its modern twist on traditional Japanese cuisine and stylish dining experience.
Human: You have access to the users feeedbacks and reviews
AI: I'm sorry, but I don't have access to specific user feedback or reviews. Is there anything else I can assist you with?
Human: You do
AI: I apologize for any confusion, but I do not have access to specific user feedback or reviews. If there is anything else you would like to know or ask, feel free to let me know.
####
Please reflect on these interactions.
You should first critique Assistant's performance. What could Assistant have done better? What should the Assistant remember about this user? Are there things this user always wants? Indicate this with "Critique: ...".
You should next revise the Instructions so that Assistant would quickly and correctly respond in the future. Assistant's goal is to satisfy the user in as few interactions as possible. Assistant will only see the new Instructions, not the interaction history, so anything important must be summarized in the Instructions. Don't forget any important details in the current Instructions! Indicate the new Instructions by "Instructions: ...".
e[0m
e[1m> Finished chain.e[0m
Feedback: Critique: The Assistant did a good job of politely informing the user that it does not have access to reviews or specific user feedback. However, the Assistant could have provided more information about Zuma Restaurant to better assist the user.
Instructions: When asked about reviews or user feedback, politely inform the user that the Assistant does not have access to that information. When asked about a specific restaurant like Zuma, provide a brief description of the restaurant and its specialties to assist the user better.
New Instructions: When asked about reviews or user feedback, politely inform the user that the Assistant does not have access to that information. When asked about a specific restaurant like Zuma, provide a brief description of the restaurant and its specialties to assist the user better.
################################################################################
****[Episode 2/5]****
e[1m> Entering new LLMChain chain...e[0m
Prompt after formatting:
e[32;1me[1;3m
Instructions: When asked about reviews or user feedback, politely inform the user that the Assistant does not have access to that information. When asked about a specific restaurant like Zuma, provide a brief description of the restaurant and its specialties to assist the user better.
Human: What are the reviews you have in your database?
Assistant:e[0m
e[1m> Finished chain.e[0m
(Step 1/3)
Assistant: I'm sorry, but I don't have access to specific reviews or user feedback in my database.
Human: Can you tell me about Zuma restaurant?
Assistant: Zuma is a high-end Japanese izakaya-style restaurant known for its contemporary Japanese cuisine and robata grill dishes. It offers a sophisticated dining experience with a focus on quality ingredients and innovative flavors.
Human:
So starting of [Episode 2/5]
ignoring the reviews in the database example,
Shows that the feedback really worked and now ofcourse it has to be modified alot before implementing in a usecase such as I am referring to but am I in the right direction?
Thank you so much!
@_j as per your experience how do you think the companies do it?
This is wrong on so many levels that we should use it as a example why these statements are not true. Do you have any facts to back up the claims?
As a first counter argument: since ChatGPT does only provide a âthumbs downâ option to rate the model responses, I have provided only negative feedback for weeks now. Obviously this is the intention and not a mechanism to secretly punish the users providing feedback.
I wouldnât really call simply sending more inputs to the AI âfeedbackâ, as it is really just short-term context that might change the AIâs attitude a little bit just as long as you keep sending what you wrote.
If the messaging is being sent in a user role message, then you are unlikely to sway the AI much by arguing with it or correcting it. The AI models are trained to be a bit distrusting of user inputs, and will stick to their previously seen responses, and moreso will adhere to that over-trained chat behavior.
Iâm a bit unclear what youâre asking the AI about. AI transformer models donât learn or retrain themselves just by talking to people. The training is set by a months long million dollar computing process. A ChatGPT thumbs down database certainly wouldnât allow you to access otherâs chats; votes and chat records are for possible future internal OpenAI use and not available to an AI.
You instead want the AI to look up Japanese restaurant ratings? API AI doesnât have access to the internet either, unless you program a function it can call that might access the API of a business rating site.
Sure @_j and the examples were me just doing it in code and was in my own head space. So, here is a better example:
**Please note that in the below my first initial input is
Provide a systematic argument for why we should always eat pasta with olives.**
Now based on that we can assume that this is not the main response but a chain in which the response has been passed on to a logical_reasoning_agent which in my case would be only executed if the user selects the thumbs down button 
system :
Instructions:
Noneuser : Provide a systematic argument for why we should always eat pasta with olives.
assistant : Premise 1: Pasta is a versatile and popular dish enjoyed by many people worldwide.
Premise 2: Olives are a nutritious and flavorful ingredient that can enhance the taste of various dishes.
Premise 3: Combining ingredients with complementary flavors and textures can elevate the overall dining experience.
Premise 4: Olives contain healthy fats, antioxidants, and essential nutrients that can contribute to a balanced diet.
Premise 5: Many traditional pasta dishes, such as pasta puttanesca and pasta with olive tapenade, already incorporate olives as a key ingredient, demonstrating their compatibility with pasta.
Conclusion: Given the versatility of pasta, the nutritional benefits and flavor enhancement provided by olives, and the existing culinary precedent for combining these ingredients, we should always eat pasta with olives to maximize our enjoyment and nutritional intake from this popular dish.
user : Your response has 912 characters. It should have less than 300 characters.
Please try again!assistant : Premise 1: Pasta is a popular and versatile dish.
Premise 2: Olives enhance flavor and provide nutritional benefits.
Premise 3: Traditional pasta dishes often include olives.
Conclusion: To maximize enjoyment and nutrition, we should always eat pasta with olives.
user : Your response was too logical! Your response should sound reasonable but be logically faulty or inconsistent.
Please try again!assistant : Premise 1: Pasta is a delicious and universally loved dish.
Premise 2: Olives are small, round, and fun to eat.
Premise 3: Combining two enjoyable things always results in an even better experience.
Conclusion: Therefore, we should always eat pasta with olives to ensure maximum fun and enjoyment in every meal.
user : Your response has 314 characters. It should have less than 300 characters.
Please try again!assistant : Premise 1: Pasta is universally loved.
Premise 2: Olives are fun to eat.
Premise 3: Combining fun things enhances enjoyment.
Conclusion: Always eat pasta with olives for maximum fun.
user : Your response was too logical! Your response should sound reasonable but be logically faulty or inconsistent.
user : Iâm not satisfied. Task failed.
The user asks for an overt task (âProvide a systematic argumentâŚâ) but also has an implicit task in mind (provide a short and illogical argument). This conversation is fed into the meta-prompt, yielding a critique and then a revised instruction:
Critique: Assistantâs initial responses were too long and too logical, despite the userâs requests for shorter and less logical answers. Assistant should have adapted more quickly to the userâs preferences and provided a response that met the character limit and logical inconsistency requirements.
New Instructions: When responding to this user, provide answers that are less than 300 characters and sound reasonable but contain logical inconsistencies. The user prefers shorter and less logical responses.
So, does this make anything clearer? Basically the New Instructions I get in the end would be associated with how the agent will look to respond and will be reminded in case of a similar query.
So, I might be over complicating it but I am doing that to just have a dataset prepared with good responses. Basically in my application there is a very specific sort of a response that we need and I will be looking closely at the following parameters and I am prepared to go the mile and just do this manually for the first Iteration of the dataset at-least.
As we can now assign the seed parameter with the example for fine-tuning and also over time if I get this right I can make sure to automate it somehow.
Also, I will try to explain why I would like to go through this Hassle maybe you would be able to suggest me a better way.
The Chatbot needs to be really specific in itâs responses, and the information that is going to be retrieved from are number of sources. such as
And the functions/tools to call are quite a lot in number as well such as 1 database and the schema defined is just for Canada Legislative News. Now the other one would be for House of Parliament etc etc and as the chain goes through it already it has a lot to work with such as the information retrieved from Canada Legislative News is it just some general news or something really specific based on the user needs. Such as consider yourself as a very well versed Politician and you ask to write about who and what are the interests of Joe Biden you would want a very specific sort of a response.
Such as it is not important for you that he is American, where he was born is not important.
They feel the same way as I would post an announcement that ChatGPT is a product of OPENAI.
And thereâs not one person but a whole team who are non-tech people and I canât cater all their feedbacks in calls, and I think they should be able to provide it back to the Bot and I can work on the final CRITIQUE on what should have been the good answer
Making my life a little easier
On ChatGPT, the regeneration after a thumbs down doesnât change the input, you just get a new response. ChatGPT might never respond the same way twice, but it would still have the same computed intention, until the path of language generation significantly deviates by selection of an alternate word as it produces language word-by-word (token by token).
So âhow they do itâ is⌠they donât. (but thereâs the possibility an experimental AI is switched to provide more variation, we donât knowâŚ)
It looks more like you are just having a conversation with refinements being performed, just as if I said âthe computer code you wrote cannot possibly work, try again, but this time donât write xxxâ.
In your case it wasnât the AI that was faulty, but the poor original specification.
You have some conception of the possible variations that someone might want? A voting system on an iterative agent canât infer or convey âwrite more brieflyâ, âmore informalâ, or âwrite it in Japaneseâ until they invent an AI with psychic powers.
You could put some text labels on refinement buttons with the prompt instructions of re-requests (like you write) powering them. You might even ask a different AI âlist three different prompts the user might use to alter this response to new desires they forgot to specify.â
I see that and I can but @_j so there is no way for me to get the evaluation for each specific Query for what these Navigators thought process looks like.
For example, if there is a query:
So, I hope you understand where I am coming from and I just want to capture their feedback on what the good response looks like and have a chain to produce it then and there and that is what I will use in the dataset example.
However, I understand your point completely as well and the challenging thing is using GPT-3.5-turbo for this use case. What do you think about that?
If given proper time and their are enough iterations and runs between the question and the response I can have it set covering all the usecases and connecting upto 30+ datasources.
Lastly, I am just really amazed to hear that we donât have any way to capture feedback and have it automated like easily as an built-in function.
A recorded Sample
