Hello, I’m trying to teach to a Custom GPT a new programming language. Reason: I noticed that the official ChatGPT Chat 4.0 is not enough skilled in easyLanguage, a niche script-based language used to develop trading systems.
I have experienced a lot of syntax errors, lack of library knowledge, and in general a lot of hallucinations in the standard chat. So a custom GPT to help me develop quickly in EL was a good decision.
The issue: I spent hours in the chat to pass all my detailed knowledge, style and tactics, and the custom GPT apparently updated the model when necessary or when explicitly asked by me. And apparently it made a lot of progress when chatting in the last days.
I was shocked when today I realized that it forgot everything and started again to answer in a generic way, ignoring all the conversations and updates (including the uploaded documents).
Question: the only thing customizable is the description and upload documents (that often it ignores). This is really what a custom GPT can do and learn? Or there is different strategy I have to use? Question is: chatting with a custom GPT is fine-tuning or not at all?
Notice also: it was labelling his code with the easyLanguage word (see picture). Then at some stage it ignored and labelled EL code it as pascal again.
It seems you’ve created a customized GPT model and attempted to enhance its knowledge through direct interaction. However, this approach might not be effective, as GPT models don’t retain information from previous conversations. Each new chat starts without any memory of past interactions.
A more viable method would be to upload documents containing the knowledge you wish to impart. This should enable the model to reference and recall this new information. While I’m not certain if this is the best or only method, it’s definitely worth exploring.
Yes I have uploaded a document in addition to the prompts. But searching in a document is not the feature that I was looking for. I had another idea of custom GPT advertised as specialized knowledge GPT, but done without coding.
And yes sometimes the chatbox said that cannot recall previous conversation. But more often that not it said that updated the knowledge and the behavior.
It works like that: You will see a wrench icon rotating, and at the end it confirm the update.
I also asked explicitly and it said that it stored the information in the internal framework.
A lot of people intended and tested a customGPT like me. Here an example of a very specialized GPT.
From what you are describing it appears that at some point you were able to get all the necessary knowledge to write in that language into the context window.
But when the token amount was exceeded the model lost track of what’s important to you and reverted back to standard mode.
First of all, when you get a reply that does not match expectations, reject it and regenerate a few times. Make sure to never keep any reply in the history that does not meet expectations. Maybe the conversation could have been saved, or maybe you still can.
Second idea is to try a new conversation and push the knowledge into the context window in a few rounds of conversation. I’d assume you should be able to recreate the previous experience for a few more replies before exceeding the context window again.
I usually try to avoid these situations whatever possible but would be interested to learn if this will work for you.
Thanks I will try. We need to know if the “knowledge” retained is only on the uploaded documents. Because in this case the chatbot said to me that it can upload up to 50MB, and it’s a lot of words/tokens.
However conceptualization is different that mere text in file (that potentially the engine may not even consult).
So what is the real implications of this “updating” mechanism?:
the updating mechanism simply updates the instructions area.
Instructions is the only way to communicate some customization and concept. The rest of the knowledge can be put on files and uploaded.
So basically at the moment a no-code Custom GPT is a search tool on your data with some element of rebranding of the original ChatGPT.
But it does not do fine tuning. If it has a “wrong” idea of something is keeping it, there’s no way to change its “mind” outside the context of the session.
It seems that it’s learning inside the session, but it’s an illusion, all is lost after. Of course you cannot put in the instructions all the details of real specialized model.
I will investigate the more advanced approaches, using code to do fine-tuning.
I faced similar problems. The only advantage that I found was using the RAG feature to reduce hallucinations. In fact I struggled to get consistent output, especially when I tried using web browsing feature. I created a financial expert, that was directed to access earning releases from web and then analyze them. But GPT consistently failed to extract the information, at times making up the information. Finally I ended up loading the earnings of companies that I want to analyze and made a very specific bot.
The only way I found apparently is to tell GPT, in the instructions area, to check the documentation before answering questions. This works, not all the time, and force GPT to review the docs before answering. However expecially on coding GPT make up things for scripts languages it did not know. I think fine tuning is the way to go.
It’s worth mentioning that if you were in the “preview” window engaging in chat, this is both not persistent nor updating the gpt. It’s more the sandbox where you are testing the build. I say this because I’ve totally been there, making great headway with the gpt design within the preview gpt not realizing that I wasn’t actually updating the gpt itself.
I’m not sure if this is your situation, but thought I would point out the distinction that chat in preview window doesn’t seem to persist – its more a demo – and so if you refreshed the browser after a few days of preview conversation, it’s “memory” would be lost.
I hope this is helpful, and that I haven’t totally misunderstood. Please let me know any thoughts on this!
I was using the “design” chat on the left, not the preview chat on the right. It was updating apparently because it shows the update icon. But as stated before in my experience it was updating merely the instruction text. Custom GPT are more new “characters” than anything. Things may have changed now, however OpenAI should specify what to expect and how it works.
Just be aware if you have input a lot of instructions, but then try to tweak them using the dialogue window, all the instructions can disappear and be reformatted by the bot. I always manually save all my instructions before I tweak them (learned the hard way)
