I am trying to figure out if there is a difference in the way models understand conversation history if is passed to API messages field vs provided as context in system prompt. I assume there is magic woodoo done to messages. Which option would yield better performance?
Welcome to the community!
I’m operating under the assumption that this isn’t really the case.
There are some issues with the model being oblivious to the last assistant message in some cases, so I wouldn’t end a messages array with an assistant message, but other than that, it doesn’t really have any special meaning, other than that it represents how the models have been trained.
I suspect that you’d only really run into theoretical, maybe performance issues with very short queries if you didn’t use the messaging framework at all and just sent raw text without the requisite beginning tokens, but the API won’t let you do that.