Is it possible to have the responses from the API fit inside of the max_token limit, so that the responses are not cut off at the end? Currently I have a max_tokens set, and I am telling the API through the system role to answer in one to two sentences. I have been raising the max_token limit, and the amount of responses getting cut off got better. But is it possible to have the responses fit inside of the max_token limit?
So that, if I want responses in a 60 token limit, GPT phrases and responds, so that the sentences fit inside that limit?
I have also tried using other system role prompts, such as “Keep your answers short.” or “Keep your answers very short.” but it hasn’t helped much.
Any feedback is appreciated, thanks.
GPT 3.5 is a lot worse at following such system message than 4
But both can be trouble to keep making short responses.
What somewhat worked for me was to instruct GPT to put some special sign like pipe | after 2 sentences and then make that a stop sign
About stop: OpenAI API
Perhaps it would help to write few example messages where after 2 sentences you mention |
(I didnt try giving example messages)
But its probably not 100% either
Another thing I did, was just to find the last . or ! or ? in the completion returned and cut the string after that.
This way you get only whole sentences and any sentence that ended cut off is removed, because it didnt end with any of the end of sentence signs.
Then you also only include this cut string in any history/memory.
You will waste some tokens, but ultimately this is currently the best way for me to only have whole sentences when max_tokens is reached and sentence is cut off.
But to actually make it do really short sentences, not quite sure how.
(I wasnt really trying to do short sentences much, I mostly only needed max x sentences)