Let’s say I save the user and assistant messages and pass that in the body while having a new prompt.
What will be the actual prompt for the API? Will it be the previous saved conversation + the new prompt or only the new one?
The actual “Prompt” is the unseen <token>assistant<token> after the messages you send, that gives the assistant the prompting to write as itself after seeing a stream of other messages.
A completion model otherwise would just continue completing the text you input. If there was just a stream of words, it might write more as you.
You can consider everything sent to be loaded into the context window as “context”. In chat models, that is loaded into the ChatML format that contains messages.