Memory for Chat Completions

API
1

I am not sure and because of that I would like to ask a question about Chat Completion. I have been using chat completion for over couple of months, and I have encountered with something new. Inside a new chat (via API calls) if I continue on the same topic (normally this was not possible) with making new requests; my ChatResponse ID appears to stay same. It was changing on every request. Does this mean that API has memory now?

I tried to send a new request without sending previous messages. seen that memory feature was not working. I believe memory feature is only available with assistants but with this recent update (same chat response id) I started questioning.

Any ideas?

2

No, I don’t believe the Chat Completions API has memory yet or we would have seen a big announcement and changes to docs.

In any case they have a product that already fits that niche (Assistants)

Very easy to test this.

State some opinions like what your favourite colour is and then follow up by asking what your favourite colour is.

My Chatbot will get it right every time because it has local memory, without it, it would fail most of the time.

1 Like
3

Welcome @yournowto

Here’s the structure of a chat completion response

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "gpt-4o-mini",
  "system_fingerprint": "fp_44709d6fcb",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "\n\nHello there, how may I assist you today?",
    },
    "logprobs": null,
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 9,
    "completion_tokens": 12,
    "total_tokens": 21
  }
}

The chat completion id in my tests has always varied with every single request.

ETA: I can confirm that it’s still different for every API call @yournowto .

2 Likes
4

My solution also has memory but in the end you need to send previous conversations to API which causes high tokens. (Talking about chat completions)

For assistans things are totally different

5

Yes previous I was also getting varied id’s. But in the recent versions it started to give same id’s interestingly.

6

You get what you pay for :wink:

It’s not like using the Assistant API will save you money, as I suspect the token usage will be even greater.

7

Interesting.

Then what is your suggestion?

8

Reduce your context window to the minimum practical, use RAG and use cheaper capable models like mini.

Also: monetize accordingly!

1 Like