I would like ChatGPT to summarize answers to a questionnaire for me, but it’s giving me really bad/unusable responses which I can’t seem to improve on.
I’m trying something like (redacted some aspects):
FROM SYSTEM:
You are a helpful assistant that summarizes responses to questions from a questionnaire that was distributed to xxx. The questionnaire aims to assess xxx. Your responses must be accurate, concise, and you can omit thanks. FROM USER:
The question is: Does your xxx do xxx?
The list of possible options are:
Option 1: Yes
Option 2: No
Option 3: Maybe
80 responses where received. The answers were:
Respondent 1: Yes
Respondent 2: Yes
Respondent 3: No
(etc.)
I’m getting results like:
Out of the 64 xxxs that responded to the question, 52 xxxs (81.25%) [do it]. 8 xxxs (12.5%) do not [do it]. 4 xxxs (6.25%) responded with “maybe” or “sometimes”.
That doesn’t include all 80, and it doesn’t even match the results, which are:
Option
Count
Maybe
7
No
11
Yes
62
Is ChatGPT not (currently) capable of doing this kind of summarization accurately, or is there a way I can improve the results?
I know you are using the API and plugins are only available with ChatGPT web interface.
The Wolfram plugin might help.
I would have tried it for your question and posted a result but there is not enough data to do a meaningful example. Also ChatGPT with plugins is only available from the web interface so the prompt would need to modified accordingly.
You could also try modifying your code for the API and make use of an external app such as Wolfram.
If you post an updated prompt I will try it using the ChatGPT Wolfram plugin.
Thanks for your offer, Eric. I can’t give the real data out as it might be sensitive, but here is an example that should have a similar output. I’ve tried on ChatGPT, and it’s a little more acceptable, but not much - it still has errors. I think it’s better because I enumerated the organisations, whereas in my API call I use a code for each “organisation”.
You are a helpful assistant that summarizes responses to questions from a questionnaire that was distributed to a number of organizations. The questionnaire aims to assess their capability in various areas.
The question is: Does your organisation require customers to pay with a credit card?
The list of possible options is given below:
Option 1: Yes
Option 2: No
Option 3: Maybe
80 organisations responded. Their answers were:
Organisation 1: Yes
Organisation 2: Yes
Organisation 3: Yes
Organisation 4: Yes
Organisation 5: No
Organisation 6: No
Organisation 7: Yes
Organisation 8: Yes
Organisation 9: Yes
Organisation 10: Yes
Organisation 11: Yes
Organisation 12: No
Organisation 13: No
Organisation 14: Yes
Organisation 15: Yes
Organisation 16: Yes
Organisation 17: Maybe
Organisation 18: Yes
Organisation 19: Yes
Organisation 20: Yes
Organisation 21: No
Organisation 22: No
Organisation 23: Yes
Organisation 24: Yes
Organisation 25: Yes
Organisation 26: Yes
Organisation 27: No
Organisation 28: Yes
Organisation 29: Yes
Organisation 30: Yes
Organisation 31: Yes
Organisation 32: Yes
Organisation 33: No
Organisation 34: Yes
Organisation 35: Yes
Organisation 36: Yes
Organisation 37: Yes
Organisation 38: Maybe
Organisation 39: Yes
Organisation 40: Maybe
Organisation 41: Yes
Organisation 42: Yes
Organisation 43: Yes
Please analyse/summarize the answers that were given.
Using the prompt as given here is the ChatGPT completion.
Based on the provided responses, it appears that the majority of organizations require customers to pay with a credit card. However, without the responses from all 80 organizations, a complete analysis cannot be provided.
From the responses of the first 43 organizations:
33 organizations (or approximately 76.7%) responded with "Yes", indicating that they do require customers to pay with a credit card.
7 organizations (or approximately 16.3%) responded with "No", indicating that they do not require customers to pay with a credit card.
3 organizations (or approximately 7%) responded with "Maybe", indicating that the requirement may depend on certain conditions or circumstances.
Please provide the responses from the remaining organizations for a complete analysis.
Here is a modified version of the prompt to get it to use the Wolfram plugin, I added a pie chart for pizzazz.
You are a helpful assistant that summarizes responses to questions from a questionnaire that was distributed to a number of organizations. The questionnaire aims to assess their capability in various areas.
The question is: Does your organisation require customers to pay with a credit card?
The list of possible options is given below:
Option 1: Yes
Option 2: No
Option 3: Maybe
43 organisations responded. Their answers were:
Organisation 1: Yes
Organisation 2: Yes
Organisation 3: Yes
Organisation 4: Yes
Organisation 5: No
Organisation 6: No
Organisation 7: Yes
Organisation 8: Yes
Organisation 9: Yes
Organisation 10: Yes
Organisation 11: Yes
Organisation 12: No
Organisation 13: No
Organisation 14: Yes
Organisation 15: Yes
Organisation 16: Yes
Organisation 17: Maybe
Organisation 18: Yes
Organisation 19: Yes
Organisation 20: Yes
Organisation 21: No
Organisation 22: No
Organisation 23: Yes
Organisation 24: Yes
Organisation 25: Yes
Organisation 26: Yes
Organisation 27: No
Organisation 28: Yes
Organisation 29: Yes
Organisation 30: Yes
Organisation 31: Yes
Organisation 32: Yes
Organisation 33: No
Organisation 34: Yes
Organisation 35: Yes
Organisation 36: Yes
Organisation 37: Yes
Organisation 38: Maybe
Organisation 39: Yes
Organisation 40: Maybe
Organisation 41: Yes
Organisation 42: Yes
Organisation 43: Yes
Please analyse/summarize the answers that were given
* Use Wolfram plugin
* Create a pie chart
* Display numbers on pie chart
Here is a pie chart summarizing the responses from the 43 organizations:
From the chart, we can see that:
- The majority of organizations (31) require customers to pay with a credit card (Yes).
- A smaller number of organizations (7) do not require customers to pay with a credit card (No).
- A few organizations (5) are uncertain or have a variable policy regarding credit card payments (Maybe).
If you want me to try some variations on this to get a better reply please ask.
The reason I am demonstrating the Wolfram plugin here is because many here really don’t get to see plugins in action and can only guess. At least this way they can get taste of what is coming and be better prepared when the plugins are generally available.
I have it working on open-ended questions, too - that was actually my starting point. I haven’t yet checked the accuracy of those responses. I thought I would try it on the closed-ended questions, and getting these inaccurate results. I’m not sure if that translates to the open-ended ones also then being unreliable - I still have to go through a few of them and check. I’d assumed it could handle this kind of task, but maybe not yet.
That’s great, thanks for doing that, Eric. I am on the waitlist for the plugins, so I look forward to playing with them, but it ultimately needs to be done via the API, as it would be part of an application’s offering.
Your results were accurate though - i.e. 33, 7, & 3. Does WA improve the accuracy, or was WA just for the chart?
Wolfram plugin only did the chart, ChatGPT did the text summary which was wrong.
In a few earlier examples that I did not post where Wolfram returned text results it is more obvious that both Wolfram and ChatGPT are generating separate parts of the response.
I do not know if ChatGPT working with the Wolfram plugin could seamlessly combine elements in say a single paragraph where Wolfram did the number evaluations and ChatGPT layered in the natural language.
I’m actually generating an interactive chart myself in the webpage, so I don’t currently need that, although it is very cool that it can be done. I really just need an accurate write-up of the answers. As mentioned to Jake, it seems that in it’s current form, ChatGPT is not ideal for close-ended question analysis.
