RAG input via System message: JSON vs plain text

I think @merefield is talking about an agentic approach (a-la assistants)

but you seem to be pre-fetching documents the good ol’ way.

notes on the system prompt

fundamentally there’s no real difference between a user, system, or assistant message, although OpenAI likes to pretend like there is (for “safety” reasons). Overall, your entire conversation is just a document.

What the “system prompt” may or may not do, is shape the model’s focus. If your system prompt gets super long, the bottom part of the system prompt likely doesn’t benefit much from the fact that it’s in a system prompt.

notes on the format

Regarding format: some folks seems to recommend structured markdown, (mostly headers I think would be important) you can try if that works better with mini. But with stronger models, I don’t think it matters all that much.

notes on “Is it OK”

Everything’s OK as long as it works :laughing:

Most of these structures are just artificial anyways. Underneath it all is just a completion model that doesn’t really care about these ‘rules’, but might be trained to behave in specific ways. How it actually acts can deviate from what OpenAI intended, so there aren’t any real hard guidelines here.

3 Likes