I use SQL and it’s not a mess at all.
Let me show you:
Here is my SQL (working) scheme I currently use:
create_table "chat_conversations", force: :cascade do |t|
t.string "description"
t.integer "message_count"
t.string "user"
t.string "messages"
t.integer "total_tokens"
t.integer "prompt_tokens"
t.integer "completion_tokens"
t.datetime "created_at", precision: 6, null: false
t.datetime "updated_at", precision: 6, null: false
end
I store all the messages in a single string , which is the serialized messages array. This is the “standard” way of storing JSON and other data which requires serialization. This method is “as old as the hills” as they say, going back 15 years at least and is still used today.
In Rails, we set this up easily in the model as follows:
class ChatConversation < ApplicationRecord
serialize :messages, Array
end
Let’s create a chat, for example and look at the DB:
First we create a new chat and strip all prior chat message which were stored in the DB:
Set Up Chat
Completion
Let’s look at the DB now:
You can see that the chat messages are all nicely serialized in the DB.
This is the current “state” of the chat.
Now we can run another chat, of course. I pull all the messages from the associated chatid from the DB and add the new message, for example to illustrate:
Second Chat, Use Prior Chat Data from DB
Chat works as expected, system role is correct:
Lets go directly to the DB:
You can clearly see that all is working fine. The system message works fine as well (debunking the community and discord misinformation that the system role does not work at the beginning, BTW. It works just fine if the session management code is written properly).
The DB now has serialized 5 messages, 3 from the first chat API call, and 2 from the second.
Of course, as the number of chat messages increase, we start bumping up against the very annoying 4096 OpenAI token limitation, so we have to prune.
In my lab, I current test a variety of “pruning messages” as you can see in the menu below:
We can use the TikTok tokenizer to estimate token amount and add this to the running token count in the DB to prune the messages before sending them to the API and then update the DB with the new messages array. This is only one of many possible pruning strategies. Currently, I disabled that feature and enjoy pruning manually because that is how I develop full situational knowledge of what is going on with the new API method.
What I am trying to mention is that it is pretty easy for a web developer to use a DB to manage the chat API dialog, store the messages array in the DB and to prune the messages array as required (based on your favorite pruning strategy), update the DB, rinse and repeat. This is all “very basic” webdev 101 stuff, to be candid.
Storage is easy. Pruning is a bit more tricky, but not so difficult with a bit of practice and testing.
Also, note, as mentioned before, there is some misinformation in our community about the “system” role not working as the chat progresses. This is only true if you drop the system role from the messages array (which is not a good idea if you want to keep the role active).
However, if developers manage the messages array properly, the system message work fine because the system role remains in the messages array.
Hope this helps.
