Greetings my friends.
Conceptual question here and seeking out who has experience using “developer messages” in a significant way.
Will provide context and then my question.
[CONTEXT]
In my particular use case I’ve got a relatively refined system at this point for managing API calls to the LLM and building a context window based on a given set of “conversation” data.
At this point, the conversation data is relatively dynamic, involving document sharing, orchestration/agents, assigning “types” to messages, etc.
I’m moving forward with exploring a 3D context window framework, so that instead of sending a string of “conversational messages” back and forth, that I’m dynamically building a single role: developer and role: user message that gets passed to the LLM in a call.
When in a certain mode (orchestrator mode) I’ve set it up so that my system, receives the response from the API and simply goes through a process of “updating the context window”. For example the response is provided from the LLM with structured data that subsequently is processed as an “update” to various "types of data within the context window.
I.e. the LLM response “updates” the docs portal, the agent calls/status, the [GOALS] the [DESIGN] the [TASKS] etc.
The system then turns around and simply calls the LLM again with the updated context window.
[QUESTION]
I’m trying to decide what’s best (of course I’ll end up trying both pathways, but I’m curious as to others experience) in terms of using developer messages.
I’ve noticed already that these work “much better” (or rather much differently) than “system” message used to, before “developer” message were introduced.
The specific question is this:
-
I’m already using the “developer message” to pass top-level data about the system (i.e. it’s not going to be linear, don’t use recency bias, etc.)
-
Can I use the developer message further as I would a “normal role: user prompt” in the sense of:
- Fleshing out “profile content” that is specifically related to exactly how I want the model to perform in the given context. For example, sometimes I’ll be pursuing coding tasks, but sometimes I’ll be performing document review, etc. This data will change dynamically as assigned on a “per-thread basis”.
- I’ll also be passing various levels of instruction about exactly what kind of structured data has to be returned (not conforming to existing specifications like pydantic or json, this is customized structures based on essentially blocks of specially delimited text) for various situations.
- Finally, I’ll be giving sets of logical guidance about how the model should actually operate in the environment (for example, recursive checkpoints, “stepping back and analyzing the context from a certain perspective after a certain trigger is processed” etc.
-
Is it most appropriate and will it be successful to wrap all this kind of data into the “role: developer” message, and then allow the block “role: user” message to function at the lower-level of actually passing in the documents, the live-updated information from the agents, etc.? Or is the developer message processed under a special set of circumstances that is going to limit it’s effectiveness?
-
Overall, I’m just not able to find reference to the use of the developer message in this level of detail anywhere I’ve looked - is it processed essentially at the same level as a normal prompt but at the “higher override level” as described in the documentation? Or is there other limits /processing directives that are going to interfere with complex and long-winded developer messages that wouldn’t occur if the same data was passed as role: user?
Notes:
-
I’m using the completions API (I guess it’s old school now!)
-
Yes, I’ll be pretty much exclusively using the reasoning models class that support developer messages for this aspect of the system, though perhaps in some settings I would be using 4o models so I guess I would need to “include the developer messages within the role: user message” if I want to achieve reasonably similar results with those models…
-
Yes, the system is setup (and I’ve tested this to some extent with great success already) to no longer provide a string of role: user and role: assistant messages. Instead, every single API call will be passed as a single role: user message, that contains the sum total from the backend of the “activites that have occurred” as a “live/dynamic 3D context window representing the assistants progress moving through a process of automatic system-generated API calls”.
Spoiler alert - it works great - you don’t have to provide turn-based context to the LLM - but you do have to make it clear about what your doing so that it can attempt to handle a context that’s often a bit different than it’s training data…
Thanks to any and all who can provide input.