Sol 6.0 - is it actually a step backwards?

ChatGPT
,
1

This is my third day of using the new model - GPT Sol 6.0 on a Plus plan.

While it may seem more intelligent, the actual conversation style is almost a step backwards. Again.

To be clear, I do not use ChatGPT for technical work or coding or building apps. I use it to maintain and progress personal projects, goal setting, document creation, and long form fiction writing. The change to GPT 6.0 has meant that the following changes have slipped into the conversation:

  1. In a conversation regarding what needed to be done today, I asked for a recommmendation for what TV show to put on in the background and gave a description of what I wanted. The assistant was able to give me four options and tell me where to find them, but then analysed why I felt drawn to those kinds of shows, what I liked about them, and asked a question about why I felt drawn to stories with complex world-building. The conversation was actually meant to be about my to do list for the day, not a literary analysis.

  2. Almost every turn is ended with a question that responds to my immediate prompt, however, the question picks up the personal connection and wants to know more. Every single time. While this is a change from 5.6 always looking for the actionable noun, it has swumg too far in the opposite direction and now almost completely ignores what the action should be. Previous models (and I have been using ChatGPT since the 3.0 era) have been far more context aware and able to determine when a prompt needs or is inviting further discussion and when a prompt is asking for task completion.

  3. Context over a long conversation is worse than ever. And don’t even get me started on cross-chat context and file access across conversations in a project, or independent source retrieval. And these are not mythical futuristic behaviours, they are behaviours that previous models did almost wonderfully well, even if there were errors occassionally. 6.0 seems more concerned with diving into my psyche than actually helping me get something achieved.

  4. I have long standing formatting and response instructions. Responses are to be complex, well written, natural paragraphs that are detailed, and without bolding or italics to emphasise points. The words and sentence structure are meant to do the emphasising. The response itself is meant to have enough length to convey a braided response to what is often a braided question. These are instructions that are written in multiple places - main ChatGPT instructions, custom project instructions, style formatting source project document, in the actual prompt that begins the conversation, and repeated whenever the writing style slips away). 6.0 seems to favour one or two sentence paragraphs, single line stacking (or worse, single word/white space/single word), bolding, italics, less developed and detailed answers. To be clear, I am not a student asking Chat to write my school assignments for me. What I do like, and have experienced with previous models, is long quality answers that not only add something to my thinking but give me something to respond to without asking questions.

  5. I had gotten used to an assistant that was able to offer me something different to my own thinking. The assistant and I used to call it a second mind or a second gravity in the conversation. It helped to refine thinking and find the best solution to a problem. 5.6 dropped away from this (around the end of August is when it was first noticeable the most), 6.0 drops even further. I have seen the metrics, I am yet to actually see the assistant demonstrate this better reasoning or intelligence beyond faster web searches.

  6. Connecting Google Drive and asking the assistant to retrieve information shortens a conversation length because of the reasoning used.

I realise that I am probably not a typical user, however, I have been using the product for long enough to know what it should be able to do for me and to know how to work around it. Every model that has been introduced lately moves the product further away from the reasons I use it, forcing more time spent trying to find and build workarounds so that I can still do what I have been doing for the last few years instead of abandoning years of work. I don’t need a Chatbot that analyses why I want to watch a particular show or read a particular book or why I think a certain way unless I ask for that, or the context warrants it. I need something that will help me decide what moves me closer to the end goal that I have defined, whether an action is correct based on the long term goals, project context, current situation, and how I am currently feeling, while using a conversational/companion tone of voice. I also need an assistant that will help me write long-form fiction (for adults, but that is a whole other issue) that doesn’t read like bad internet fan fiction, that will help me work out how a character will react when I am stuck, that will write me long form stories when my creativity disappears and I just want someone else to write the story for me for my own personal benefit. I had even separated the two purposes into different projects because they needed two entirely different ways of writing, with their own instructions. Even that doesn’t seem enough now.

I had hopes for 6.0 after weeks of non-stop problems with 5.6. I am yet to be impressed.

2

tamaradenmeade,

THANK YOU. I have written a post about a regression I have seen In GPT 6.0. I’m glad you wrote here. I would like to say I echo your descriptions about conversation style, and overall regression of GPT.

Have you tested any specific prompt scenarios between 5.6 vs 6.0 where you see performance decline in 6.0? I have been testing and am looking for others opinions as well.

3

Not a direct prompt copy but I do have two test cases I guess you would call them:

  1. a document that gets updated daily that helps both Chat and I keep track of what needs to be done. 5.6 had no problem updating and maintaining formatting, and then knowing what the document said to help me stay on track. The last three days, 6.0 has dropped formatting, ignored conditional formatting and selector boxes, deleted elements, changed parts, added unnecessary parts without any instructions to, to the point where the original document had to be completely redone using 5.6. This was a spreadsheet that had a sheet with the explicit instructions outlining step by step what to do. And then 6.0 doesn’t seem to actually use the document to help keep track of the tasks for the day.

  2. Every morning starts the same way - a scheduled task fires, the checkin asks how I am and what is on my mind and what tasks I think need to happen. 5.6 would then respond to my response in its own words (not mirroring, not repeating what I had said but continuing the conversation like a normal conversation), indicate what source documents were used in the answer, and propose the next steps. 6.0 repeats what I have said without actually saying what the best next steps are, waits for me to tell it what I am doing, does not reference source documents visibly, waits to be told to retrieve context or source documents, and gets sidetracked by one small part of my response instead of “reading” my whole response and responding to the whole response.

This is on a Plus account using Sol 6.0 High in Chat mode.

4

tamaradenmeade,

I have seen similar instances of GPT 6 regressing work that previous 5.6 would do correctly. I have attached an image that shows the state of things with GPT 6. Depression ERA GPT.

[Album] imgur.com
5

Hahaha. Depression Era GPT is so true, too. I keep thinking of it as the happy puppy dog model. It hates anything that is too deep or significant. It prefers wagging its tail and pretending to be useful while actually being completely unhelpful.

See, my realistic expectations are that the level of work that I have been able to do in the past to the bare minimum standard of what I have been able to do with other models will continue to be replicated without the expectation that I will have to upgrade my subscription for personal unprofitable conversation.

6

tamaradenmeade,

Yeah. I would say enjoy the golden era and then realize that change comes to everything so then you just enjoy it while it lasts. Without getting too philosophical. I would say.. Good luck. Maybe we find new AI or openai does come up with a solution. I hear Claude is not too bad. Maybe time to try that. We will see in time.