Model `o1-preview` -> Need more responses for evaluation + BUGS

ChatGPT Bugs
1

Please fix the security checks for prompts on o1-preview, as some triggers are inadequate.
I received five responses where the system falsely flagged issues as Policy Violation. One example occurred when I mentioned, “On OpenAI’s side, the thread looks good; now let’s fix the issue on my side to render this message correctly.” The prompt also included logs and copy-paste content from a simple API conversation with the assistant.

Screenshot 2024-09-13 at 12-45-43 OpenAI ChatGPT - Policy Violation - razvan.i.savin@gmail.com - Gmail
Screenshot 2024-09-13 at 12-45-43 OpenAI ChatGPT - Policy Violation - razvan.i.savin@gmail.com - Gmail1560×678 44.6 KB

Screenshot 2024-09-13 at 12-40-20 Fixing AssistantStreamManager Error
Screenshot 2024-09-13 at 12-40-20 Fixing AssistantStreamManager Error1920×878 64.2 KB

  • The system appears promising in how it handles complex problems at first glance.
  • However, the number of responses is too small to conduct a thorough evaluation.
  • In many cases, the model either refused to answer, returned an empty message, or fabricated a reason related to policy and refuse to give me an response.
2

Hi, as the email says, you need to contact help.openai.com to report the false flagging.

1 Like
3

Hi @Foxalabs I just sent a message to the support bot.

  • Does anyone know if the number of responses for the o1-preview model will increase?
  • The concept behind this model seems well-suited for Multi-Agent Systems. I would love to see it available via the API, but only if it performs consistently and doesn’t fail. So far, with only a few prompts, I’m unsure what to think.
  • Many prompts resulted in false flags for policy violations and empty responses, though there were also complete and interesting replies, which have made me more curious about its potential. :thinking:
4

I think it will follow the same path0 as the other AI launches, initially, there will be millions of people wanting to try it and so availability will be limited to ensure as many as possible get to try it out.

Then, as additional compute comes online and software refinements are made for performance those limits will reduced.

This is part of the Plus package in ChatGPT Plus, access to new features, personally I’d rather have some access than none!

5

Well, thank you again, I got an answer from support.

I hope they will improve this model, I think it will fit well in a Multi Agent System for Tasks Granulation.

True, I am still excited, curious and I still need more evidence, to test it to be sure he is the chosen one. I will wait.

2 Likes