I have built several GPT’s that have DALL-E capability enabled. Recently, after a long-form chat session (but not really that long), the GPT claimed that it could not analyze images.
“While I can’t view images directly, based on your description…”
I also had a GPT claim that it could not reference prior portions of the same chat session. For “privacy reasons” it was unable to “recall” even topics only a few responses back.
Very odd behavior that I am unable to explain.
I have not experienced this behavior with regular ChatGPT sessions.
It’s as if the chat session got corrupted somehow over time, within the space of maybe an hour. No amount of discussion or analysis could deter it from it’s new understanding of its capabilities.
Starting a new chat session with same GPT resets its capabilities but the prior session appears to remain broken.
The longer the conversation gets, particularly if you have a lot of similar content, it becomes increasingly harder for the model to perform its job
Also, it depends on how ChatGPT is actually built. In my experience with gpt-4-1106-preview (on the dev/api side), a well worded user message at the bottom of the conversation trumps a system message at the top.
It’s quite possible that ChatGPT gets better over time, but simple chat implementations do suffer from these issues.
When you notice that your conversation starts deteriorating, it might sometimes be a good idea to ask for a summary and paste that into a new chat to “continue” where you left off.
Also, it’s often better to edit a prior message and regenerate an answer than to repeatedly get the model closer to a certain answer.
I agree with ChatGPT losing context over time in a chat session, but I’m referring to it’s basic capabilities. It would be like ChatGPT “forgetting” that it can search with Bing during a regular chat session.
yep, developers often inject function or tool instructions in the system message, which is typically prepended to the top of the conversation thread.
so over time, even that gets lost
It’s a usability problem for sure, but the longer the conversations get, the wonkier things become with such a simple system.
IF you’re motivated i’d urge you to explore the API. It’s not as easy to use, and you’d need to integrate your own browsing tool or use bing chat directly, but it might be worth your while - you’ll have a lot more control over all aspects of the model.
This is a GPT that I built using the GPT Builder. The instructions are not complex and make no effort to curtail abilities, protect the instructions, etc.
It may have been misleading to mention that it was a long-form chat. I meant to imply that it was not simply 5 or so exchanges back and forth. But it was far less than what I typically consider to be a long-form chat session.
In the session where it forgot about DALL-E, there were no other indications of losing context. All responses were relevant, contextual, perfect recall of the entire chat, etc. Only thing missing was DALL-E-related capabilty.
With regular ChatGPT, I have never experienced even an extremely long chat session losing it’s ability to search, generate images, or use code interpreter.
I have explored the API, and while that’s a different and viable option, I’m just curious about this particular behavior of a “GPTs chat session” and if I’m the only one experiencing this behavior.
Unfortunately can’t help you further here beyond the generic make sure you mention the capabilities in the instructions etc etc, I hope someone else chimes in!
No worries, thank you for chiming in. For now I’m chalking it up to GPTs growing pains/scaling issues/beta feature/etc.
I find GPTs to be a nice balance between basic ChatGPT sessions and applications built with the API. I often start with ChatGPT, move ideas into GPTs, then if they seem viable and worth greater exploration/sophistication/time commitment, I move to the API.
I can’t get it to retain much at all. I ask it to write all replies in UK English, for example, then a few messages later it’s using US spelling again. Or I ask it to avoid particular words and it goes ahead and uses them anyway.
I will draw attention to these issues during the chat and it will apologise and correct itself, but then go back the same errors.
