@daveshapautomator, actually the Instruct models follow directions that you give it fairly precisely. I achieved fairly good results trying this technique out:
Asking Davinci Instruct to provide exactly 50 word summary of how macaroni was invented:
Total word count: 41
Max tokens allocated to it: 539 (Well over the 50 word limit)
Here’s another:
Provide a 70 word summary of how the French war happened:
Total word count: 65
Max tokens allocated to it: 703 (Well over the 70 word limit)
Important Note Regarding Using Numbers with GPT-3:
Okay so I just realized something that’s important to note here. While these models are good with following instructions, it seems like with higher word count limits, 100+ word count limits, the models actually don’t follow instructions and I think I know why.
I was able to achieve remarkable results with lower numbers but not with higher numbers. This was the same issue I encountered in my first post regarding whether GPT-3 is capable of being good a mathematics, where I learned that it performs well with algebra when using low numbers such as 33 or even 77 at times, but when tasked with bigger numbers exceeding 100, it started to actually perform worse and I believe it has something to do with how the models are trained on natural language which is different from the numerical form of numbers.
I discovered that the word form of numbers proved much more effective when communicating with these models, which I explain in great detail in the post mentioned above.
Failure point when tasked with providing exactly 177 words describing how advertisers lure people into buying their products:
Total word count: 89 (Well under the 177 word count)
Max tokens allocated to it: 1521
This is when I learned that converting numbers from the numerical form into word form proved way more effective, as seen in this next example:
Total word count: 194
Max tokens allocated to it: 1521 (Well over the 200 word limit)
If you’re seeking higher word count limits, try using the word form of the number and also just to help the Instruct models out, you can certainly emphasize that you’re wanting a specific word count in your completions!
Here’s a similar prompt where instead of using 200 words in word form, it’s in numerical form:
Total word count: 474 (Had to stop the model from generating more words)
Max tokens allocated to it: 1522
I also had a tough time getting Davinci Instruct to follow instructions when tasked with generating paragraphs with a certain amount of sentences. Usually I’m pretty good at prompt engineering to get desirable output quickly. I’m seeing that the Instruct models are quite sensitive to the starting text you use alongside your prompt. I found that after several tries, I got Davinci Instruct to almost achieve the 5-sentence long completion after I used ‘Short Report:’ as the starting text. However, I also instructed Davinci Instruct to provide no more than 5 sentences but then got 6. It does seem like prompt engineering as well as the starting text carry significantly higher weight when generating completions.
Hope this helps @jefftay and everyone else!!