Custom gpt vs assistant api

API
21

I’m also running into this issue. Does anyone know what might be causing this?

Here is my use case: I created a custom GPT and an Assistant designed to write SQL code using an information schema uploaded as a CSV file.

This is the instruction I gave to both:

You are an SQL developer. I will ask you questions and I want you to write SQL code that produces the answers to those questions. I want you to use the attached information schema to write this code.

It works without problem on the custom GPT, but doesn’t work on the GPT Assistant. On the Assistant, I get a message saying it needs a schema (doesn’t see the CSV file, which does work on the customGPT), or it gives code that doesn’t reflect the schema (e.g. it uses some of the right tables/columns, but also some that aren’t in the schema).

Two options come to mind to me:

  1. The custom GPT uses a different version to the GPT version available to the Assistant. The Assistant API is using gpt-4-1106-preview, and I can’t see what version is used by the custom GPT.

  2. The custom GPT gives different instructions (perhaps a “pre prompt”?) that isn’t seen by the user.

The Assistant API also seems unable to interpret a database information schema (saved as a CSV file), but the custom GPT can.

I hope this helps!

22

I am on the same boat. Custom GPTs vastly outperforms the Assistants built with OpenAI platform. They both use the exact set of instructions and the same files as context information.
I think we need more clarity on how each of them operates so that we can better understand and assess when to use one or the other.
Having access to the GPTs through an API would also be a good solutions.
Can we get any kind of feedback from OpenAI itself?

2 Likes
23

I would like to share my experience on this topic, maybe it will amuse someone, or someone might advise me.

When I tried CustomGPT, I was amazed by its sensational performance, its understanding of the context of information, and its excellent adherence to instructions. I immediately looked for an API integration option and found links saying, “hey, create your own customGPT, just like the Assistant API, it’s the same thing.”

Okay, I sat down, looking forward to my own AI that would cover corporate helpdesk, and wrote a knowledge database, which I tested with CustomGPT (220 pages of text, about 1200 questions and answers, explanations, etc.). How did customGPT handle it? Brilliantly! It precisely connected answers from different topics, was able to answer new questions with original correct solutions and precise correct procedures.

I tried giving the same knowledge database to the Assistant. The result? Horrible. Each query consumed 7000 to 15000 tokens and the results were inaccurate. Where customGPT gives the correct solution and procedure, the Assistant replies, ‘I don’t have information about that in the source data,’ or totally hallucinates with a non-existent procedure.

After those tens of hours, I decided not to give up. The next step to achieve my own AI helpdesk = fine-tuning. For fine-tuning, I used the knowledge database of questions and answers, created alternative versions of the questions for the validation dataset. I ran the GPT 3.5 model (unfortunately, a better version is not available) through fine-tuning four times, with the final result of Training loss 0.1258 and Validation loss 0.2629. The result? Fine-tuned ChatGPT 3.5 responds completely off, unable to answer any “moderately difficult” question correctly. The level of hallucinations is like a hippie on the third day of a rock festival.

I returned to Assistant AI and studied the embedded knowledge database, thinking that in this structure it might search more efficiently and understand more information with less token consumption. I converted the knowledge database into an embedded format – from a 0.6MB txt file it became a 150MB json. It can’t work with such a large file. I divided the json file into smaller batches, but there are more than 20 of them, which exceeds the capacity for uploaded files for the Assistant API (and my current knowledge database covers only 30% of the information).
Either I made a rookie mistake, or it’s simply not worth the effort right now, and it’s best to fold my hands and pray to OpenAI to release an API for customGPT, because none of their other models achieve the same results with the use of a personal knowledge database.

I would like to be wrong. Please, if anyone knows, advise me.

1 Like
24

I’ve seen a lot of people reporting the same (Custom GPT retrieval being better than Assistant Retrieval).

The truth is we don’t know really what’s going on behind the scenes. Assistants hasn’t received any sort of noticeable update for 1/4 of a year while ChatGPT is being updated/modified all the time.

So it could be that they are actively experimenting and gathering metrics through ChatGPT usage, and then eventually will update the Assistants once they have established some methods.

Yikes, yeah. Fine-Tuning is for behavioral adjustments. Not ideal for knowledge, for many reasons.

In the meantime I’d learn about embeddings, vector databases/knowledge graphs.

1 Like
25

It seems like it has something to do with the Temperatures and the possibility of having a custom instruction enabled in ChatGPT.

We don’t know if the temperature of CustomGPT and Assistant API is the same.

26

I think part of the reason for this is that OpenAI needs to keep ChatGPT competitive.

They essentially give us the same or similar backend framework through Assistants API, but I suspect their “cutting-edge” retrieval resources and methods are gate-kept, or at least prioritized for ChatGPT.

In my opinion, Assistants API is still worthwhile due to access to Code Interpreter and a self-managed tool calling system. But for RAG and retrieval in general, I think Assistants would do much better with a custom solution.

27

Sometimes they almost seem like two entirely different models! The custom GPT model appears to have less flexibility but a higher quality, whereas the assistant model has more flexibility but a lower quality. I wish they were the same, but they don’t seem to be! :frowning:

28

I´ve been working with the GPT Assistant for the past 45 days. I was able to set many different and important boundaries as well as making it respond correctly based in a very small database. The trick I found to make it work is to divide the instructions very precisely and also use separate files for each type of task you want it to perform. So for example, you can set a list of tasks that it must follow (task 1, task 2, task 3 and so on…) and for each specific data analysis you tell it to “read file XYZ.csv” (or whatever) that contains a specific content or decision tree inside. It takes a while also to get a grip on how to use the perfect prompt… I had 3 different Assistants with the same purpose, but sometimes with different prompts, and a very small change in a word (like <strictly state “X”>) can impact in the performance of the result.

4 Likes
29

Would you mind to share your instructions? Thanks!

30

It´s something like this (to give you the idea):

“Chat interaction script you must follow:”
"1. Use the input and read the file “XYZ.csv” - this file should also contain straight instructions on what you want to do, like a decision tree, for example, that you may use as an output for your chat (as a response)…
“2. Next, read file “ABC.docx” and state the options for the user”.

In general, the idea is to use 3 things, as I mentioned:

  1. ;
  2. ;
  3. ;

PLUS doing that in a very coordinated roadmap. You will have to test and adjust to find what works best for your purpose.

Hope this helps! Cheers!

1 Like
31

Sorry… 1, 2, 3:

  1. instructions;
  2. inputs from the user;
  3. files with other short instructions or short (and) direct (and specific) knowledge base;
32

This is I also thought about, any plans to implement this?

1 Like
33

two complicated and goes against general LLM-idea to work naturally with information…

34

I can only confirm this, until today custom retrieval with the same parameters and sometimes even without any, outperforms assistants in any way, shape or form.

35

True that. An updated version that I´m using now is to set thresholds and let it flow more freely + using function calling to interfere within a thread. This way I even managed to get rid of all files (of course my project doesn´t deal with huge file data) - which became faster and cheaper to implement. Other AIs around have different specific capacities as well… so I totally agree with you, but it´s definitely permitted to use a LLM in such sense if you´re counting on its understanding of user´s requests (again, it depends on the project´s objective).

36

can it be done now? i have already had ‘credentials’ from my custom GPT in GPT store. how can i connect it to my separate app with assistant API. thanks

37

Today is March 31, and with the exact same setup. Custom GPT performs significantly better than assistant API.

I specifically picked out questions from the dataset and custom GPT was able to answer it and follow instructions well. However, assistants API wasnt able to do so.

Here is the prompt i used
You are an intelligent customer support agent, and you have been provided historical customer chats and company documentation. Use the provided context to answer each question in 1 paragraph or less, with the answer in the first sentence. Avoid any sales-speak and avoid explaining acronyms. Avoid mentioning that you’ve been provided company documentation or community discussions for assistance. If no context is provided, you can say you don’t know and another agent can step in.

Assistant sometimes can’t get it in 1 paragraph, or end up explain acronyms. But the biggest problem is that it cant find the data in the files. So it cant really answer questions.

At this point I dont think assistant API is top priority for openAI

2 Likes
38

From last week(the 3rd day of openai 12days live show), I notice that the assistant API outputs reach the same level as custom gpts with the same settings, and becomes more reliable and comprehensive than before. I didn’t change anything with my setting ( file_search data, functions , etc.). Maybe the openai service down changed something. : P Is there any one has the same findings like me?

39

Hey, do you still experience these improvements? Because I just tried it out and nothing changed for me…

40

My Personal Opinion on the Appropriate Usage of Custom GPTs vs Assistants API vs Chat Completions API

  1. Custom GPTs - All-in-one / Ready to Use / App-Packed
  • Preconfigured and optimized for conversation-type interactions.
  • Have built-in thread management with conversation context.
  • Include built-in tools like web search, image generation, vision, voice, etc.
  • Can connect to external APIs (via tools).
  • Cannot be called from outside!
  • Allow the use of knowledge files for specific domains, but you have limited control over how these files are used in context retrieval.
  1. Assistants API - Chat Completions API with Extra Capabilities
  • Supports persistent conversation threads (you need to create and manage threads manually).
  • Can access stored files and utilize OpenAI’s internal vector store for context retrieval (but you do not have direct control over this vector store).
  • Offers more control over model parameters (but you are responsible for testing and configuring them properly).
  • Can connect to external tools via function calling (you must configure and handle these connections).
  • Can be called from outside.
  • The built-in vector store makes it easier to manage context for ongoing conversations, but it comes at the cost of limited flexibility compared to custom vector DB solutions.
  1. Chat Completions API - Stateless Remote AI Functions Accessible Over an HTTP REST API
  • Offers full control over model parameters and context retrieval.
  • Is completely stateless (no built-in support for persistent threads, unless you build your own mechanism).
  • Supports external tool usage via function calling (with complete freedom to integrate with any system or database, including custom vector DBs).
  • Can be called from outside.
  • You can hook it up to any external vector database of your choice (Pinecone, Weaviate, etc.), providing maximum flexibility for managing context and knowledge retrieval.

Summary

As a rule of thumb, the more control you give away to OpenAI, the more general-purpose and ready-to-use the solution will be. If you need precision or specific configurations, the Chat Completions API will offer the most flexibility. For simple, out-of-the-box setups, Custom GPTs are the quickest way to deploy something useful. The Assistants API sits somewhere in the middle, offering more built-in features but with some trade-offs in control.

Vector Store Management

One important nuance to consider:

  • Assistants API: Comes with an internal vector store for managing conversation context, but this vector store is managed by OpenAI, and you do not have direct access to it. You can upload files, and the assistant will use them to retrieve relevant context automatically.
  • Chat Completions API: Requires you to manage your own vector database if you want to use one. This offers more control and flexibility, allowing you to use external solutions like Pinecone, Weaviate, or even self-hosted options.

My Personal Approach

When building a solution using these tools, I follow a step-by-step process:

  1. Quickly draft the system and user prompts in the playground to test the concept or task.
  2. Analyze the task to break it down into smaller subtasks, identifying opportunities to parallelize or simplify processes.
  3. Craft and test each subtask in the playground.
  4. Set them up as tools behind an API gateway to access them via REST API calls.
  5. Get the API definition with all tools documented.
  6. Pass the API definition along with a "master’ system configuration to a Custom GPT for quick interaction testing (most projects never go beyond this step, as it is often good enough).
  7. Review interactions through the Custom GPT to see if a full app (with a database, UI, etc.) is necessary. If not, stop there.
  8. If persistent threads or complex context management is required, decide between Assistants API and Chat Completions API based on the level of control needed.

Common Pitfall

I have seen several cases where developers chose the Assistants API for its built-in thread management, only to realize later that their application required more control over context retrieval. This often results in them converting to the Chat Completions API, which could have been a better fit from the start. The key takeaway: do not rush the decision—spend time analyzing your application’s requirements before committing to a specific API.

Hope that helps. (AI edited)

2 Likes