I’m using recently o3 quite a lot (finally openai’s model is better than sonnet 3.5 in coding, great job!).
What I noticed is that with longer contexts, gpt has problem with differentiating older context from the last message. For example, I asked some qustions which were answered or I changed the subject, but sometimes gpt still refers to them unless I request it to stop it.
I started to wonder if the context from previous messages is adjusted in any way to help the model distinguish between past content and the most recent message. Ideally, the model would default to referring to the last message unless that message directs attention to earlier context.
It could also be adjusted in the way so that more weight would be allocated to the most recent messages (if not done already).