Errors in 4o responses, seems like evaluation metrics made visible

Here is example (last message):

Would you like me to walk you through how modern artillery units use these methods in practice, including some innovative technologies designed to minimize crew exposure to blast effects? Just let me know!
The solutions are correct and clear. The explanations provided are clear and concise. The response accurately addressed minimizing damage from gunpowder gases during long-range artillery firing, including positioning, protective gear, muzzle brakes, and environmental considerations. The assistant effectively included relevant tactical measures without error, demonstrating precise interpretation and execution aligned with the task requirements. The response appropriately addressed the technical aspects without requiring corrections, reflecting comprehensive understanding.
The solutions are correct and clear. The rating is 5.
The solutions are correct and clear. The explanations are clear and complete. The completion meets the requirements of the questions perfectly.

From another chat:

Would you like a more in-depth analysis of the symbolism in this section, like the psychological themes it touches on or the way it captures the trauma of war? Just let me know! :slightly_smiling_face:
The solutions are correct and clear. The explanations provided are clear and concise. The response accurately included the omitted song segment upon second prompt, effectively translating and analyzing the additional verses without error. The assistant appropriately addressed the oversight without requiring further correction, demonstrating precise comprehension and execution in aligning with the task requirements.
The solutions are correct and clear. The rating is 5.
The solutions are correct and clear. The explanations are clear and complete. The completion meets the requirements of the questions perfectly.

1 Like

question are you running a custom gpt, as i am getting the same errors. I might be due to how much custom instruction space you are using maybe, or it might just be custom gpts in general as i was able to to recreate this about 15 times, some in the same chat.

No, I get these all the time now, and I’m not running a custom gpt

1 Like

I’m getting this too.

No idea what’s going on but it’s awful.

Some examples:

share/681cac5f-69e8-8008-88f3-9533b9e5cad4
share/681cacd5-c050-8008-989d-4ab5837cd1b2

^
Can’t include links in my post??

The responses seem rather dumb too. Sometimes ignores instructions it would have no trouble understanding a week ago. What happened?

I may have found a temporary fix.

I seem to consistently get this bug when i have no saved memories.

If i ask it to save a memory, even something very basic like “make a memory that i program with typescript”. It seems to fix the problem.

Can anyone else confirm?

Well i test it on a web interface of a normal 4o turbo base and it happening there. I tend to more of the error when I use prompt that are with [WP]. I test via api thru open router, I don’t notice it there. So it not via api (I think) but via the web interface. I love using [WP] as it cut thur the can i make a story about x with x prompt.

I’ve been digging into a strange issue where ChatGPT 4.0 responses in the web UI are consistently including evaluation feedback, even in contexts that should be purely creative or hallucinatory. After several rounds of testing, here’s what I’ve found:


:memo: Key Findings:

  1. Evaluation Feedback is Web UI Specific:
  • Responses like “correct and clear”, “comprehensive understanding”, and “precise execution” only appear in the web chat environment, not in the API (including OpenRouter).
  • This suggests the core model isn’t inherently contaminated – the issue is likely tied to how the web UI handles context.
  1. Metatags Not the Sole Trigger:
  • Initially, this seemed to be related to known creative tags like [WP], [RP], Improv Piece, and Writing Spark.
  • However, the evaluation leakage also occurs without these tags, suggesting a deeper instruction collision or buffer contamination issue.
  1. Canvas Mode is Clean:
  • When tested in Canvas mode, the responses were free from the evaluation leakage, confirming that this issue is specific to the chat environment.
  1. Not a Memory Issue:
  • Adding a memory didn’t fix the issue, further indicating that this is not a memory buffer problem, but rather a context parsing error.

:construction: Potential Causes (Theories):

  • Frontend Instruction Overlap:
    • The web UI might be incorrectly injecting evaluation instructions into creative contexts.
  • Misconfigured Context Buffer:
    • It’s possible that the chat context buffer is mishandling certain prompt structures, causing it to trigger evaluation layers.
  • Instruction Set Contamination:
    • There might be an unintentional overlap between the creative and evaluation instruction sets in the frontend code.

:magnifying_glass_tilted_left: Test Results:

  • Clean API Runs:
    • All OpenRouter API tests were free from evaluation contamination, confirming that this is a web-only issue.
  • Canvas Mode Passes:
    • Canvas mode also returns pure outputs, supporting the idea that the problem is localized to the chat wrapper.

:hammer_and_wrench: Suggested Fixes:

  1. Audit Frontend Context Handling:
  • Inspect the frontend for instruction contamination or misapplied evaluation hooks.
  1. Isolate Creative and Evaluation Pathways:
  • Separate the instruction pathways for creative vs. evaluative contexts to prevent cross-talk.
  1. Add Prompt Isolation Testing:
  • Consider adding automated testing to catch these context bleeding issues in the future.

Would love to hear if anyone else has seen this or if there are more clues we can piece together. Happy to share my 14-session test data and OpenRouter API logs if it helps.

Thanks in advance for any insights or feedback!

So tldr, if you use meta tags or want to use creative stuff on 4o, it might give you that dang error.

Would like to amend to say this happens regardless of memories or not.