Tldr is the last paragraph at the bottom.
I understand where you are coming from and what you are saying. This is a developer forum and everybody who visits here regularly likely has the skills to perform these tests with ease and considering the good faith some members of the community are bringing towards this truly amazing tech we should provide hard facts with no problems whatsoever.
On the flip side many of us have access to GPT4 via a paid subscription only, you know the Plus thing. People subscribe and remain paying users because the experience is great. Period. Suddenly these users from various corners of the net start reporting issues. Having a marketing background I know exactly what that means. Consider that the people reporting issues have been using the product for months and have been paying because it’s worth it. The same users suddenly report that it’s not worth it anymore and the reports are accumulating. I call this a hard fact. Still people with a different background may differ on this. I understand and accept that.
But let’s focus on the data collection process. Of course I can dig into my past conversations. Different conversations regarding different projects start with the exact same prompt and I would get a comparable answer every, single, time. This has changed. I can now dig deeper into my data export, collect all the answers and pinpoint the exact date this occurrence appeared for the first time and how this kept repeating for about a month now. Then pack it up nicely and send it to a random OpenAI team member without expecting any answer whatsoever. Which is fine. My dataset is small, unstructured and in no way representative. Furthermore I understand that the community managers are executing this role with a low priority due to immense workloads.
[Edit: removed because it’s unnecessary]
So, where does that leave us now? And what does it mean for the discussion about these observations?
What I am saying is that as a paying beta tester there is only so much additional complication I am willing to accept in order to drive somebody else’s project to commercial success. I hope this makes sense and that nobody feels offended as this is more aimed at clarifying a different perspective on this topic.