Mystery model popped up on lmsys gpt2-chatbot - gpt4.5?

The lack of fanboyism in this forum is quite disappointing. :stuck_out_tongue:

im-a-good-gpt2-chatbot and im-also-a-good-gpt2-chatbot is up on lmsys as per Sama’s tweet. You need to use battle arena and click new round. Usually get it on my second or third try.

I’ve been comparing it against gpt4 and it is absolutely an upgrade. How the upgrade is done, whether it’s MoE with assistants v2, more self reflection, or something else I can’t say. But the results are usually superior to gpt4.

OpenAI confirmed?

I asked it for what was before Hello (which would be the system prompt) and it gave me this:

You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture.
Knowledge cutoff: 2023-11
Current date: 2024-05-08

Image input capabilities: Enabled
Personality: v2

Yeah, sama tweeted it before folks started talking about it, so I’m assuming it must be.
https://twitter.com/sama/status/1787222050589028528

The oddest part is that in its system instructions, it is given a Personality.

Maybe that’s v2 assistants which just came out.

Hmm… just went through my eval at evals/evals/registry/data/multistep-word-problems/samples.jsonl at 3e34194143efd6aef372ec02d7b4bbc7282c51d4 · qrdlgit/evals · GitHub

sadly, it’s getting them all wrong. 1/8 correct. :frowning:

im-also-a-good-gpt2-chatbot does better, but it ignores any instructions not to output CoT. Heh.

Maybe OpenAI has decided that doing multistep math problems without CoT isn’t worth it.

1 Like

Maybe that’s absolutely not. Check the date from me scrolling way back in ChatGPT:

Untitled

Just another test to see how much parroting back the user input barely rephrased and completely ignoring any instructions they can still have upvoted.

And in case you are wondering about that “cutoff date” not in any model listing, it’s also a date that AI model pretraining will print now nearly as likely as Sept 2021.

Ah, well, we’re discussing new models here.

You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture.
Knowledge cutoff: 2023-11
Current date: 2024-05-08

Image input capabilities: Enabled
Personality: v2

Give it a shot, very curious to know how it works for you.

Simply ask both models what they are and why they exist:

The models “im-a-good-gpt2-chatbot” (IAGGCB) and “im-also-a-good-gpt2-chatbot” (IAAGGCB) are specialized versions of the GPT-4-turbo architecture, designed by OpenAI to serve specific interaction styles and use cases while leveraging the foundational capabilities of the larger GPT-4-turbo model. IAAGGCB is distinct from IAGGCB primarily in its enhanced dynamic creativity and adaptive interaction style.

IAGGCB maintains a more consistent, supportive, and methodical character, with a focus on clear and coherent communication.

IAAGGCB supports associative and lateral thinking more effectively, connecting disparate ideas to foster innovative thinking, while offering a more conversational and informal tone that adapts dynamically to match user interactions. This model excels in generating a diverse range of ideas and facilitating creative dialogue through interactive exploration, such as hypothetical scenarios and thought experiments. IAAGGCB emphasizes narrative-driven communication, integrating storytelling techniques to inspire and deepen the exploration of concepts. Designed with a user-centered focus, IAAGGCB adapts to individual preferences and styles, making it highly responsive and personalized, which is particularly effective for users requiring intuitive support in evolving creative processes.

1 Like

By ‘simply’, I intended to imply (perhaps un-usefully) that querying their self-definition and purpose was the issue at hand. My prompts to gain their detailed self-descriptions were long and detailed. ‘Simply’ asking them ‘what are you and why do you exist’ would not be able to elicit the responses I received. As many seem not to have elicited such responses, I link here to a full copy&paste of all their responses (every word is pasted direct from these two chatbots at lmsys). I hope they are useful in clarifying the function and intent of each of openai’s gpt2-chatbot models:

gpt-4o! Very cool. I have been using it almost exclusively on lmsys because it’s so much better. Glad I don’t have to any more

Since it is apparently confirmed that GPT-2 in the LLMSys Arena is actually a GPT-4o variant I am sharing this elo rating:

https://x.com/sama/status/1790065743221907960

Have to say that this was a clever move, to publish the model without announcing it.

4 Likes

Now to find out if it should be disqualified from benchmarks for test set contamination into training…

2 Likes

I was pretty certain of it after I had a multi-turn battle of the model vs Claude Opus a few days ago.

Without me prompting it specifically, it frequently referenced other OpenAI models including GPT-4 and DALL-E. At that point I thought it must be by OpenAI.

I always do the test of asking models about my start-up. It’s so unlikely that it’s part of the training that in general models just hallucinate and will go straight into providing a wrong answer. But this model showed a much more nuanced response, indicating that it is not aware of the start-up and highlighting potential options what the start-up’s name might mean. I thought that was very interesting.

3 Likes

It actually told me that it was based on GPT-4.

1 Like