OpenAI API (chat method): What is the best way to send questions

[dwang]

I’m wondering how everyone send the prompt to the server, along with the chat history.
What I did is to connect each question and answer and separated with \n,

The 1st prompt is sent as is, the following questions will be like this:
Q\nA\nQ\nA\n … Q

[ruby_coder]

@dwang

I’m wondering how everyone send the prompt to the server

My first draft looks something like this (still working on the final code and the chat_messages DB table stuff:

Basic Call to Chat Method:

       messages = []
       line = {"role": role, "content": prompt}
       messages << line
     
         client = get_client
         response = client.chat(
            parameters: {
               model: model,
               messages: messages,
               max_tokens: max_tokens.to_i,
               temperature: temperature.to_f,
               presence_penalty: presence_penalty.to_f,
               frequency_penalty: frequency_penalty.to_f,
               n: n.to_i,
               user: user,
               stop: stop
            })

Example Update DB

if use_db 
        ChatMessage.create(
            chatid: chatid.to_i,
            role: role,
            content: prompt,
            user: user,
            model: model,
            stop: stop,
            prompt: prompt,
            max_tokens: max_tokens.to_i,
            n: n.to_i,
            temperature: temperature.to_f,
            presence_penalty: presence_penalty.to_f,
            frequency_penalty: frequency_penalty.to_f
        )
end

HTH

[dwang]

Can you pass an array to the API? I’m wondering if there will be an error. I only pass a string which contains all of the chat history.

[ruby_coder]

@dwang

m wondering if there will be an error. I only pass a string which contains all of the chat history.

You must pass an array to the messages param just like the API docs show.

HTH

[zhihong0321]

Hi Ruby,

if we need to pass previous prompt into next API call ( to preserve the context of conversation ),
does it meant we will quickly hit the token limit? no?

2nd Question, is this the method we break through the single call token limit :

Much appreciate your input.

[ruby_coder]

@zhihong0321

does it meant we will quickly hit the token limit? no?

Hi @zhihong0321

You will need to write code as a developer to manage your “conversation” messages and to trim or summarize your content as you reach the max_tokens limit.

Yes, this is a current limitation (4096) which I guess will be raised at some point in time in the future.

I am on the way to the gym soon, and I’ve almost finished the first draft adding this chat method to my lab, and here is how I manage it:

• Assign a “chatid” to each new chat sessions or continue from a prior chatid.
• Save each chat completion as a new row in the DB, which creates a new chatid if this is a new chat, or use an existing chat it if an existing chat was selected.
• Include a “description” column in the DB row so we can remember what is each chats is about and use that along with the chatid to select a prior chat session.
• Add columns for usage to each DB row (for chat content) and calculate that for each session and use that number to determine how to trim, manage the session as you get close to the max_tokens limit.

To be honest, I will not finish this last piece until later tonight or tomorrow as I’m heading to the gym soon to get pumping up my muscles, haha.

@zhihong0321

2nd Question, is this the method we break through the single call token limit :

I’m not sure about this as I have not implemented the steam option today for chat, but will later on.

HTH

So far, for me today, only this part is done (below). I need to tweak a bit and finish the dialog / session management part later. Off to the gym…