Has anyone done testing with the newer reasoning models?
I find things change fast and I had trouble finding best practice for complex GPTs.
I initially had my instruction block point to my reference files to extend the instruction set but I found out that GPT won’t always load those files right away / depends on context it’s inferred.
Just wondering if anyone has done testing of scenarios (file types, instructions, linked instructions, actions etc..) for what best drives context, workflows and reference material.
Okay, cool. It wasn’t quite the angle I was asking about, but it was still an interesting read.
Funny enough, I’m already applying many of those concepts, with the exception of mandatory retrieval in every case. I’ve built an intermediary layer that acts as a control plane.
What I was really looking for was guidance on designing the most efficient GPT architecture.
What I’ve noticed/been guilty of, is setting up Projects with instructions to assist me with the design of the GPT itself.
But despite occasionally stating that I want to focus on ideal approach for LLM consumption, not human consumption, I still get some results and designs that are clearly logical for a human but not actually as efficient as they could be.
I’m on my third partial redesign, where I am going to essentially use actions to populate context and rules since I found out that actions are always “known” to the GPT and therefore proper descriptions will allow the GPT to “react” much quicker which will solve the occasional issue of the GPT not loading a workflow resource file (for example).
My current implementation had instructions set (8000 chars) but in that it would have Intent Routing with instructions like
For documentation authoring see workflow-documentation-authoring.md and while that works pretty good, it appears as though possibly having something like “getDocumentationAuthoringWorkflow” as a action may get me more consistent results.
As in, instead of “I can’t do this right now” → “Are you sure” → “Oh, I can in fact do this”
It should go straight to “Ok, here is how we do this”
But this is all assumptions and hypothesis, and I understand I have GPT that can perform many different (but related) tasks but I will only go route of GPT per intent if I have to, I think this can work since there’s some enrichment from having all of these live together since it give better picture of the overall ecosystem.
I tell the model in the instructions it has to hit the external knowledge first every single time like a wagon wheel, search the index or call the action, pull the nodes, reason only over what came back, and if there’s a gap it has to go around again. No answering from training data when the files or actions can answer instead. That’s the whole thing. Off-the-shelf model, no fine-tune, just discipline on the usage side so the graph does the remembering and the model only does the talking and the walking.
we all find our way
It don’t need to remember everything just how to find it.. like a librarian
Edit after reading your response.
is this what you are asking about?
hard reinforced actions. that’s the whole move. you don’t leave it up to the model to decide whether to go look. you write the instructions so the action or the knowledge hit is mandatory and it has to do it every single time, same path, same order. reinforce the repeat until the shortcut is gone. wagon wheel again. search or call first, reason only on what came back, if there’s a gap go around one more time. no training-data answer allowed when the action can fire.
that’s the efficient part. short instructions that force the loop, actions that are always known so it doesn’t do the “i can’t → are you sure → oh wait” dance. one gpt, multiple related intents, but every intent still has to spin the same way.
one thing though. any custom gpt you build inside chatgpt right now will break role or identity locks as a safety feature. it won’t stay in character or hold a hard mode if the system decides it needs to step out. the reinforcement still works for the retrieval loop, but pure role persistence gets clipped.
Sorry my point is simple without the forced loop, you’re still relying on the model’s judgment about when to look things up, which is the original failure mode. I wish you much luck.
Also I am just a user too lol. I just enjoy custom GPT a lot. Everyone here is just a user unless they have a mod or staff tag. It is a community that helps community.
And welcome to the forum!
While it would be annoying if chatGPT break role, it’s fine, all the actions in the middleware with Oauth so i can handled that portion of the enforcement. As long as I can feed the workflow rules and the contextual data at the moment I’m happy.
I don’t mind if it breaks role as long as it does not forget function. I have been experimenting with putting instructions into instagram with a blue check and loading a custom into a stateless UI like a new ChatGPT chat using in chat search calls.
Not with a custom GPT from OpenAI like the ones in the ChatGPT store. If asked or pushed close to a no no they will drop role, but of course can’t (most of the time) forget onboarded knowledge and instructions
Because this is something outside of our control, it sounds like you can’t fully prevent it, but again, really specific scenario would lead to that.
You can likely have your actions payload re-inject/remind of the role to reduce drift but I haven’t tested that sort of stuff.
@PandaPi This is a little structure you can use as a reminder for your AI if you do not want to use middleware.
Have the main instructions explain what each notation or note means, which knowledge source it points to, and when the model must return to the instructions instead of guessing or continuing from memory.
The notation does not need to contain the entire rule. It only needs to tell the model which rule to recover and where to look for it.
Read the governing instruction, interpret the command’s attached control notation, follow the required retrieval path, execute using what came back, and preserve that rule when the workflow reenters.
That way, the AI does not have to remember everything. It only has to remember how to find the right instruction, what to search for, and when to go back around again.
Also have the knowledge sources include back-notes that point the AI back to the governing instructions. That way, the instructions point to the right knowledge, and the knowledge points back to the rule that controls how it should be used.
I’m so weird, I always feel like I must acknowledge folks I read so I run out of emojis fast on the forum. Because I read and heart past stuff so I know I have read it lol. And I do not forget to go back and give hearts owed LOL. I’m mental
Ok so.. I though I would post now that we’ve moved forward with in this aspect. Honestly I feel like I can step back a bit now that I have completed some of those core changes.. I feel like I’ve been glued to computer lately.
Anyways, the Context Map using Actions approach is working out really well so far. It seems GPT is always aware of actions. What it’s solved:
Much faster response times
Gated Knowledge: No more folding files into reference files to overcome 20 file limit (in fact, might do away with reference files entirely, other than for some static indexes)
Because of the Gated Knowledge approach we have less drift and no limit on the data. (ps. I just realised this is just a RAG with routing, but I’m new to this so here I am re-inventing the wheel)
Context Map (In GPT Instructions, but also fed back to GPT on intent change detection)
Type
Domain
Intent
Key
workflow
documentation
lookup
workflow.documentation.lookup
workflow
documentation
author
workflow.documentation.author
workflow
asset
lookup
workflow.asset.lookup
workflow
asset
update
workflow.asset.update
etc…
Routing rule:
classify domain + intent from the request, match ONE key
above, call getOperationalContext(‘key’), retrieve only the exact key
it returns. Never guess a key. Never load unrelated domains.
Two-step pattern (discovery vs. retrieval — the LLM decides which)
include_content=false first when the key isn’t already certain — narrowing an ambiguous request, or checking for a collision before proposing a new key.
include_content=true once the key is confirmed, to pull the actual content. Most calls go straight to true; the map is usually enough on its own to know the exact key.
Anyways, this is exciting and hopefully stable enough to not have another “aha” moment that requires re-design.
I don’t have the whole context here but I would be careful about this type of rule where you are a little bit too instructive, which might prevent the model from taking a local decision. Make sure your overall instructions help the model to make a decision following your goal values to preserve and the purpose of the application because you might end up in a bunch of files which restrict the model instead of helping it.
Which doesn’t mean that the model does not need to be restricted in specific areas. What I would do is I would load all the instruction files into the latest pro model, set up to max reasoning, and check if there are any contradictions or overly “dumbing instructions” (where you take the model for a dumb entity and do the “thinking” trying to predict all edge cases).
I wanted to post and mention I realized I said something incorrect regarding RAG, I read a bunch more and understand a bit more.
I was playing around with setting up embeddings on my external contextual data but I realised it was pointless since I already did all the work around splitting the data into context-specific chunks, but I’d love to play around with that some more in future if I have giant data sets in a custom-GPT.
Regarding reasoning, I noticed the model can take a while on the reasoning side of things and I plan on doing some tests with standard vs extended to see what impact it has, I’ve just been a little apprehensive to test impact on technical workflows but I’ll test this.
I don’t mind if it breaks role as long as it does not forget function. I have been experimenting with putting instructions into instagram with a blue check and loading a custom into a stateless UI like a new ChatGPT chat using in chat search calls.