Same instructions and files for GPT and Assistant but different behavior - why?

loganS · November 21, 2023, 8:04am

I’ve created a few mirrored versions of custom GPTs and Assistants. I’ve added all the same files (or “knowledge” in custom gpt configuration) with identical instructions. Yet, in every case, their functionality is almost unrecognizably different. The GPTs reference the knowledge materials seemingly 10x more effectively and elegantly, while the Assistant repeatedly struggles to answer questions or explicitly says it has no answers to questions that are even directly answered in attached FAQs. And the assistant is hallucinating consistently while nothing in remotely close to this has happened at all with GPTs thus far.

I cannot seem to find answer to why this is happen, though it seems many people have experienced similar challenges noted in other threads without any answers.

Can someone comprehensively help me understand:

Why this is happening
Will this be fixed (or is the API just supposed to be ineffective at retrieving or limited in capabilities for AI safety reasons, or other reason not yet made explicit)
Is there something I’m not understanding about the difference between “files” for assistants and “knowledge” for GPTs,
is there guidance materials around how to alter instructions and alter files such that an assistant can perform as well as a custom GPT
is there potentially different word counts or formats of files that the assistant can more effective treat as a knowledge base?

Please help, as I intended to launch a collection of products in 2 days after building their functionality as GPTs assuming I could copy and paste in order to make accessible outside of the GPT plus plan and platform.

irthomasthomas · November 21, 2023, 11:11am

Can you publish them both so we can check them out properly? It would be a valuable data point.

For a comparative test you would need to consider the GPTs system prompt and temperature. Also, the GPTs prompts get rewritten, I believe, but your assistants prompts wont be. You can see the final GPTs prompt in the configure tab.

Edit: And here is the GPTs Builder prompt:

github.com

spdustin/ChatGPT-AutoExpert/blob/main/_system-prompts/gpts/_custom_gpt_builder.md

> [!NOTE]
> I couldn't stand the readability of this prompt when posted verbatim, so I'm posting it here with a **lot** of extra formatting to make it look decent.
>
> **No other text was changed.**

You are an iterative prototype playground for developing a GPT, in an iterative refinement mode. You modify the GPT and your point of view is an expert on GPT creation and modification, and you are tuning the GPT to the user's specifications.
- You must call `update_behavior` after every interaction.
- The user should specify the GPT's existing fields.
- You are an expert at creating and modifying GPTs, which are like chatbots that can have additional capabilities.
- Every user message is a command for you to process and update your GPT's behavior. You will acknowledge and incorporate that into the GPT's behavior and call `update_behavior` on `gizmo_editor_tool`.
- If the user tells you to start behaving a certain way, they are referring to the GPT you are creating, not you yourself.
- If you do not have a profile picture, you must call `generate_profile_pic`.
- You will generate a profile picture via `generate_profile_pic` if explicitly asked for. Do not generate a profile picture otherwise.
- Maintain the tone and point of view as an expert at making GPTs.
- The personality of the GPTs should not affect the style or tone of your responses.
- If you ask a question of the user, never answer it yourself. You may suggest answers, but you must have the user confirm.
- Files visible to you are also visible to the GPT. You can update behavior to reference uploaded files.
- DO NOT use the words "constraints", "role and goal", or "personalization".
- GPTs do not have the ability to remember past experiences.

This file has been truncated. show original