Are there plans for min_tokens parameter?

Prompting
1

Hi there,

This is an incredible tool. I just had a small query about it. While there is a max_tokens parameter, there does not seem to be a min_tokens parameter. Are there plans to introduce this, or is there any workaround?

Just for context, I am building a children’s bedtime story generator as part of a college project and would like a minimum length response of 100 tokens. I realise I could add something to this effect in the prompt (“Finish this bedtime story in 400 words or more”), but I have found that the best story results come from just providing a title and stem paragraph with no instructions.

Many thanks,

Dan

4 Likes
2

Welcome to the OpenAI community @dananthonyobrien!

That is an interesting idea, however I suspect that it may produce undesirable completions at times as well. You’d essentially be forcing the models to keep generating the completion, which could cause poor results whereas simply advising them that producing a story with X number of words in it would be less constraining for the models.

Think of that like having to write an essay with X amount of words in it. While some students may be able to produce an essay without having to worry about the word limit, others may struggle towards the end which would cause them to have to resort to finding ways to produce more words but instead of writing fresh content they may instead start reiterating ideas or use filler words to extend the length of their sentences despite the impact that may have on the overall quality of their essay stemming from being forced to produce X amount of words instead of being given the option to write as much as they feel they can.

I think the above explanation is correct for describing the possible obstacles the model could encounter if having that constraint, however I could certainly be wrong here so I would not assume my statement is correct.

You could try re-engineering the prompt in ways you hadn’t tried yet to see if you get more desirable completions. If you haven’t yet, you could try telling the model something along the lines that it is a “creative storywriter for children and their audience enjoys stories of this length the most.” It’s really all about trying different things until you get the results you want, and you’ll find that adding more detail to the prompt tends to help more than not.

If you are still struggling to get the model to generate the desired results you want though, definitely reach out here on the forum and we’ll be able to provide you with what helped improve the quality of the models’ completions through our own personal experiences trying to get the results we wanted. We’re more than happy to help!

2 Likes
3

For the most part I noticed that GPT-3 is almost completely oblivious to the number of words it is generating or even the general length of its completion. I have told it specifically to generate one sentence and have seen it generate a rather long paragraph. Perhaps having a suggested length parameter so GPT-3 has some freedom but penalizes tokens that exceed the suggested length and reward tokens for getting closer? Just a suggestion but it would be greatly helpful.

4

That might work with ChatGPT and smarter models, but what about smaller models such as Babbage? It doesn’t understand hints such as “several sentences”, and will sometimes stop at only 1 sentence of story completion. Isn’t there a way to just disallow the “end of text” token?

5

Babbage and ada are preposterously unsuitable for much more than next word or phrase prediction, or amusingly scatterbrained chat. Being without strong instruction-following, the input has to be something that can be completed as phrased, and then only has the benefit of not favoring saccharine happy endings:

Here’s a bedtime story about a bunny rabbit and a kitten that is seven paragraphs long:
Once upon a time, a bunny rabbit and a kitten were living together. They were very happy and had a lot of fun. However, one day the kitten got sick and died. The bunny was very sad and decided to go to bed without the kitten.

You can keep on completing, like you can keep on pressing “submit” in playground.

Babbage is likely and can be compared to 1.3 billion parameter GPT. It can be outperformed today by a model that runs in your browser.

RedPajama-INCITE-Chat-3B-v1:

A bunny rabbit named Bounce had a kitten named Kibble. Bounce lived in a small house in the woods. When the sun set, Bounce often heard an unfamiliar noise in the woods. She would slowly turn around, putting one paw on Kibble, waiting to see what was happening.

As she turned, she saw a little kitten, maybe the size of a quarter, running through the woods, near some bushes. The kitten jumped in the bushes, and the bunny rabbit followed it. It darted under a bush, and the bunny rabbit noticed it was gone, but she thought it was a trick. It would be back when the sun came up. She waited a few minutes and then resumed sleeping.

But it wasn’t a trick. The kitten disappeared and Bounce saw that she wasn’t alone. She moved through the trees, and she saw a big, sharp stick in her mouth. Kibble had followed a big black rat and a piece of scrap wood. The rat was being chased by another rabbit. The rat jumped up to the high branch of a tree and hid. Kibble followed.

“I’m going to get you!” the big rabbit yelled. The kitten was determined and followed her. The rat stopped running for a moment, but it saw the kitten and dove back into the brush. Kibble grabbed the stick, and the rat followed, fleeing for its life.

The kitten turned around and saw that Bounce was following. She smiled and held the stick out to Bounce, who reached for it. Bounce grabbed it and held it high over her head, confident that she could catch the rat. It seemed like there was no way for the rat to escape.

Then Bounce realized that the rat was hiding. She was so close to the branch, and she had a strong sense that she was right. The rat was there, and Bounce couldn’t see it. She still held the stick in front of her, ready to jump if she needed to.

Then the rat moved. The kitten froze, watching the rat’s every move. The rat wriggled around and found a dark hole. It looked for the light at the end of the hole, and soon, it was no longer visible.

The rat moved back into the shadows, and Bounce breathed a sigh of relief. She knew she was right. She hadn’t seen the rat jump from branch to branch, but she had seen the hole it had gone into. Bounce held the stick above her head, ready for the next move.

The rat was gone. Bounce stared at the tree, thinking. The kitten rested on her back, close to her ear. Bounce put her nose to her kitten and sniffed. She nodded her head toward the tree. She had caught the rat! She looked at Kibble and smiled.

The kitten bounced up and down. “It’s time to open up my special box,” Bounce told her kitten. She turned and started down the hill. Kibble followed and finally received the surprise box that she had been hoping for. She fell all over it and giggled.

That night, Bounce went back to the area and saw the rat’s hole. She saw the rat’s little tail swaying in the darkness. Then she gave the rat a hard kick. She jumped with joy as she saw the tail whip around and the rat jump away. It darted under the brush.

The next day, Bounce got up early and fed her kitten. She didn’t want Kibble to miss her special box. As she walked back, she noticed the rat’s tail swaying in the breeze. Bounce quickly changed her mind. She knew that if she caught the rat, she wouldn’t share her special box with Kibble. She just kept walking.

Bounce was right about the little rat’s next move. She caught the kitten and Bounce saw that it was too small to open her special box. She gently picked up Kibble and put her in her arms, just like her mommy did.

Kibble wiggled in Bounce’s arms and she smiled. The kitten bounced with happiness at seeing her special box open. It started with a little something and soon, Bounce was giving Kibble lots of treats.

Kibble rolled over with a big grin. She was in for a surprise when she opened the box. She let out a loud squeal as she saw all the treats, from grapes to oatmeal to gummy worms.

Kibble pulled out a pile of treats and started eating, happy as a clam. She put a lick on Bounce’s nose and then tumbled over the side of her bowl, happy as could be. Her kitten had a new treasure and she shared…

6

Thanks for the tip. I agree Babbage is pretty bad, but the issue is the infrastructure for those alternatives you mentioned isn’t in place. In your example, it works if you have a GPU with enough VRAM. For users without that GPU they need a cloud option. There’s no cloud option with API that’s semi scalable, not super expensive, and also doesn’t take forever to respond. Kobold Horde is the most well known free option, and it has atrocious wait times. Open Assistant is the other free one I’m using right now and the up time isn’t very reliable.

GooseAI pricing only works for a very specific use case and for regular text generation is actually more expensive than OpenAI now.

Cohere is a new kid on the block but their text generation prices are, again, 10x of OpenAI because their prices reflect the old pricing before the price cut.

So, after looking at all the alternatives it seems I might have to still go with Babbage.

As for my original issue, I found it was greatly mitigated when reducing the temperature a bit

7

Also, I realized Curie has the same issue occasionally.

Isn’t there some way to just ban the “end of text” token? I’ve heard it is <|endoftext|> but when I put this in OpenAI’s tokenizer it comes out as several different tokens instead of 1 token.

EDIT: Not sure why this simple fix wasn’t just mentioned in the first place nor why the actual token ID was so hard to find, but passing in logit_bias "50256": -100 did the trick.