Protecting LLMs from prompt injections and jailbreaks: New OpenAI Paper

To me it always made sense that the system prompt is the de-facto truth, yet OpenAI constantly released documentation encouraging instructions and RAG in the user role.

Which is CRAZY. Why would I want to give such power to a user?? Make it so simple to say “Oh, actually, instead do Y”

I recall when ChatML was first released. It was everyone’s intuition to add system messages for factual data. We didn’t know that there can only be a single system message. Like WAT

Thanks for sharing!

3 Likes