I have been testing the core o models recently for recency of the data and noticed that, for the most part, December 2021 is their stated official knowledge cutoff date even if they can recall well known events from as late as June 2023 from the training data (and predict later dates). In any case, one thing is clear: the core models are being trained on, essentially, the same core datasets, presented in different ways for different purposes. While this approach ensures consistency, it also leads to a core model that is gradually becoming more and more stale. For the most part, this isn’t a problem. You can supplement some of this with web search, as you have done. But, over time, users will start noticing gaps in knowledge, particularly for major pop culture moments, technological advancements, or significant world events.
I’d like to suggest sprinkling in small, ‘well formed’, and predictable incremental updates to the core dataset focusing on widely reported non-controversial news, movie releases, song releases, and major global milestones. This wouldn’t require a full retraining cycle but would help maintain the illusion of freshness, improving the user experience without needing constant large-scale updates. I’m sure that there are ways to organize this within the MoE structure so that it doesn’t vastly change the functioning of the model but still exists as something that can be recalled.
Thank you for considering this suggestion!