I’m building a an essay generator. I give my users 2 option to generate a large essay or a short one. Short one should not be more than 150 words. At the end of the prompt I instruct Davinci:
“Word count should not exceed 150.” All the test responses I get are more that 160/170 words…
Do you guys know what may be the solution?
@bestbubbledev - Youtube | Patreon | LinkedIn | Twitter
You would have to set the
max_tokens parameter in the JSON request. One token is 4 characters or about 0.75 English words. So for 150 words max, you set it to 200
max_tokens. This will prevent it from going over roughly 150 words in the output. So the output is less than 150 words or less. But you can still get outputs of 30 or 50 words. It doesn’t force the output to meet 150 words if that makes sense.
From my experience when
max_tokens is limited Davinci returns the text but it just limits the number of characters.
Like if I wanted to get: “This is test”
If I limit the tokens instead of potentially returning: “test” it returns “This i”
Does this make sense?
How are you trying to get it to say “This is a test”? I can see a fine-tune giving “This i”, but I would be surprised a prompted DaVinci would do this, especially if the
max_tokens is set to something larger than the output.
I just tried this Curt. Set the token limit & this is how the last sentence ends:
It's a comedic skit featuring a couple of health-conscious friends trying to",
And even if this sentence was fully finished, the logical flow of the paragraph doesn’t end on this sentence.
So limit somehow needs to be coming from a prompt…
What is your stop token set to?
I don’t have stop token included at all
Hmm, that could be the problem.
For back and forth prompts, say starting with “Human: " and “AI: “, I have to set two stop tokens in the array like so [” Human:”, " AI:”] … note the spaces preceding in the stop token.
The las line only contains:
No space at beginning or end.
Wow I couldn’t figure out what the
stop_token was for! Anyway I don’t think this will be a problem. Right now my app is just 1-way street. User generates a text, if he/she doesn’t like it, simply regenerates… OK let me explore this further. I will reach out to customer support on Monday!
The stop token is an optional parameter that you can provide when making a completion request using the OpenAI API. It is used to specify a token or sequence of tokens that, when generated by the model, indicates the completion should stop.
By default, the OpenAI API does not require a stop token, and the model will generate text until it reaches the maximum token limit specified in the
max_tokens parameter. However, you can use a stop token to control the length of the generated output or to ensure that the model stops at a specific point.
For example, if you want to generate a short response or a single sentence, you can set a stop token to a punctuation mark such as a period or a question mark. When the model generates the stop token, it will halt the completion and return the generated text up to that point.
Please observe that the post is from January. Those asking may have found their answer in the intervening six months…