gpt-3.5-16k is ignoring much of my prompt when I provide large chunks of context in the system prompt.
The answer is a pretty obvious yes. Tokens are distracting from the importance of particular actions to be taken. Mom shouting at a child in a library vs mom shouting at a child in a Chuck-e-Cheese.
A recent study.
1 Like
AS @_j has said, the performance of any LLM tbf when understanding a large context will decrease, especially when it has to follow instructions given in the prompt.
Between my experience and others on this forum, usually giving important information in the beginning or the end is the best way for it to permeate through. For a very important instruction, you can even duplicate is in the beginning and the end. The middle is usually where I prefer to keep the samples
If you construct your prompt as a “chain of thought” or logical sequential process, then the output actually improves as the prompt length increases.