Assistants API is Killing Me

API
, , , ,
1

I’ve setup an Assistant which contains one book (a few MB in size) and one other document (a few KB in size) as references. My instruction set isn’t that large. Now that OpenAI have exposed the Threads with Token counts, I’m realizing how expensive this seems to have become. Some threads with only a few messages back and forth are coming to tens of thousands of tokens. I’ve guesstimated that the cost per message (each way) comes to somewhere between $0.07 and $0.10. That’s crazy.

Anyone else having issues? Am I doing something wrong here?

1 Like
2

That’s one of the downsides of the current Assistants API (and why I don’t personally use it) there’s no way to control cost and you could easily get to a point where you’re paying $0.50 per request.

3

Normally, this data should not be directly retrieved by the Assistant API, but rather should be embedded in a vector database and then called appropriately.

4

Change the model of your assistant. Use gpt 3.5. A request is less then 1 cent usualy.

4 Likes
5

One of the benefits of the Assistants API is that it handles the vector/search functionality for you. Otherwise, using Chat Completions with your own vector/search solution would be the way to go…

6

I’ve thought of doing that, but in some cases GPT-4 performs way better than 3.5 :frowning:

1 Like
7

Of course! That’s why I handle request with gpt 3.5 and delegate more complex to gpt 4.

2 Likes
8

Please reduce the data to the minimum necessary and limit the amount used during retrieval, which may also be helpful.

9

Easy to separate if you have complex tasks. For a conversational bot, you’re either going to have good performance with GPT-4 or relatively poor performance with GPT-3.5…

10

Assistants API determines that on its own. Again, supposedly one of the benefits of this service. There’s no way that I know of to control how much context the Assistant maintains or how much of the vector database to pull into the thread…

11

For discussion, there is not much difference between gpt3.5 and gpt4, I mean small talk.

1 Like
12

To be honest in most cases I prefer GPT-3.5 for the vast majority of tasks. GPT-4 tends to overthink things.

1 Like
13

I apologize for any confusion.
What I was referring to was the size of the data when uploading…

I understand that the benefit of an assistant lies in automated retrieval.

I thought that by limiting the data to be uploaded, it might help save some costs, but I apologize if my suggestion was off the mark.

14

It’s a good suggestion! But I’m only uploading a few MB. The Assistants API can handle GBs worth of data (supposedly). Can’t really limit it any further :frowning:

1 Like
15

For the time being, put together your own local RAG agent.

Here are some code examples:

(A fully featured, stable, plugin for Discourse, written in Ruby (on Rails) )

(A Python implementation)

1 Like
16

You’re not doing anything wrong.

As others have suggested, a less expensive LLM such as 3.5 is an option because you’re relying on RAG and don’t require as much power in the LLM.

IMHO, the most effective option would be to chunk this, import into a Pinecone vector DB (it will be small enough to be run for free) and this will substantially reduce costs while potentially increasing accuracy, depending on how conducive the embedding is to a structured document/chunking.

2 Likes
17

You aren’t doing anything wrong.
It would be beneficial if the Assistant API could retrieve data from files more efficiently.

I also need to learn more about the Assistant API.

18

The whole point of the Assistants API was to cut down on all the infrastructure and coding. I had a great system working with Chat Completions, but moved over to Assistants thinking I’d save so much time on dev. I guess nothing comes for free — save time, spend more money and end up with a slower system.

19

I’ve been using it since it first launched. It performed better initially. Performance has degraded. I assumed a new Assistant would need time to “warm up” before reaching peak performance. From my experience, it seems to be the opposite.

Let me know if you figure anything out. Thanks again for the responses.

20

I’d say:

  • less flexible
  • more expensive

(at this point)