5.6 Sol became GPT3 - it is struggling with everything

[Nekomimi]

I’m experiencing generational downgrade. Projects that were handled and maintained perfectly well for months, now became a mess. It’s struggling with everything. Making changes when not asked, not keeping the established persona, messing up simple goals. I don’t know what they did, but it’s really bad.

I use Sol High for long, complex project workflows involving large codebases, archives, many files, repeated tool calls, and conversations that continue for a long time. Until very recently, it was extremely good at maintaining the state of a project across the whole conversation.

Even during the recent outage, when tools were throwing errors, the model itself was still working well whenever the request actually went through.

What I’m seeing now is completely different.

The problems seem to be a combination of several regressions happening at the same time:

• It forgets context much more aggressively. Decisions, rules, file states, and things established earlier in the same conversation suddenly disappear from its working knowledge.
• It loses track of the active task. Previously, if a long task was interrupted because the response/tool execution ran out of time, it would clearly say where it stopped. I could simply reply continue, and it would immediately resume the unfinished work.
• Now, after replying continue, it often does not continue at all. Instead I get useless conversational responses like: “If you want, I can now do X” -* even though X is literally the task I already told it to do and it was in the middle of doing.
• It stops work prematurely without telling me. Previously it would either finish the requested work or explicitly say that it had not finished yet. Now it often does only part of the task and presents the result as if everything were complete.
• Reasoning appears much shorter/weaker. Tasks that previously caused Sol High to think and work for a long time (sometimes 40min+) are now sometimes answered very quickly, with important steps skipped.
• It no longer reliably verifies its own work. It can miss files, skip parts of an archive

Additionally internal python/executable containers are constantly broken / throwing errors and the only way to make it work is to start a new chat. Branching existing chat is not helping.

Deep research is completely broken since yesterday too. It can’t do what is asked for. Normal chat can do way better research than deep research.

I’m not even mentioning recent UI changes because it is a hot mess now. Chats have often problem with loading, there’s no sidebar with chat table of content, it is loading only part of conversation (I need to refresh every part by going up and again - it is often throwing loading errors), but model stupidity is way worse.

[lachie1]

Money saving ahead of the IPO

[danefox]

Yup 5.6 Sol has forgotten so much of my project by now that I had to stop.

[rickp050]

@Nekomimi

I think a lot of these complaints are actually symptoms of the same underlying problem.

I don’t necessarily need ChatGPT to have a bigger sidebar, more folders, more Project levels, better chat titles, or another way of manually organising everything.

I need it to remember and retrieve what we have already done together.

My conversations don’t stay neatly inside one subject. I start with one problem, we discover something else, follow a branch, make a decision, work on files, return to it days later, and sometimes the useful answer is buried somewhere in that history.

What I want is very simple:

“Find everything we’ve discussed about X.”

And X could be a word, subject, project, person, file, decision, date, place, or even a phrase I vaguely remember.

For example:

“Find everything we discussed about the MHE project.”

Or:

“Where did we discuss the idea about charging rental machines only when they leave the yard?”

Or:

“Find the conversation from around September where we discussed the technician revenue model.”

ChatGPT should then be able to find the relevant conversations and give me the context, decisions, files and branches connected to them.

That would also solve a lot of the Project and Recents problems.

I don’t always know where a conversation belongs when I start it. I shouldn’t have to stop thinking about the problem so I can decide which folder to put it in.

And when a long conversation gets interrupted, “continue” should mean continue the task, not start a new conversation about what I might want to do next.

The same principle applies to files and tools. If we’ve already established the state of a project, the system should know where we stopped and what remains unfinished.

So rather than adding another layer of organisation, I’d like to see one fundamental capability:

A searchable, contextual history of my work with ChatGPT.

Not just chat titles.

Not just keywords.

Not just memory.

The actual relationship between conversations, Projects, branches, files, decisions and unfinished work.

The information already exists.

I just don’t want to have to remember when, where, what I called it, or which Project I accidentally put it in.

I’m beginning to think the biggest missing feature isn’t better filing.

It’s continuity.

My reply . …

[Nekomimi]

I’m putting all work on hold. I’m working on a game translation project where GPT is helping me figure out file formats, create font tables, keep track of terminology, etc.
Now, all of a sudden, it’s changed the entire color scheme of the game to a color that wasn’t even mentioned in any of our discussions. What’s happening right now is some kind of unfunny joke. First reasoning models two years ago wouldn’t have even done something as bad like this.

Something like this literally never happened before.

[danefox]

Exactly. I’ve used mainly OpenAI models for a year now, and this is the worst I’ve seen continuity since early 5.x models

[marydolphin2020]

The major companies are going to lose out over time to companies that balance intelligence and predictability with profit.