What you can do is do it in two parts, giving a word count/token count of the response and asking it to revise it up or down. It works ok with proportions, so if you give it current, target, current/target, and a direction I bet you could get real close to exactly the right number of words within 2-3 calls.
Any luck? One thing I’d suggest, seeing your code, is to just move all of the user instructions into one system message, which should end with “the user will provide a set of keywords with which you must follow the instructions.” then in the user message you would give nothing but the keyword list (as a python array, i do think it does better with beginning and ending tokens for things like lists).
Keep in mind that to the bot, the “system” message is neither from the user nor visible to it. You can use this to your advantage by repeating instructions in both while the system message tells the bot to listen to the user as well, but really it’d probably be best to just put all your instructions into the system message. Thats where instructions are meant to go, where the models are trained to be most receptive to listening. Also, generally chats should go back and forth between user and assistant. I havent really tried having a long string of user messages, but I could see that causing problems too.
As others have mentioned, it doesnt know how to count really (though it can predict sequences of integers, so if you wanted, say, a list of items, you could tell it to stop after some number of list items while telling it to keep track of the items with a number at the start of the list item). So, you could for instance give it the length of the message that contains only keywords and their brackets, and let it work at least knowing how many tokens correspond to those keywords. But you’ll never get exactly the length you want with a one-shot.
Ok so I was getting a few too many “just do different SEO” responses and lost track of this thread, but I got it working with a combination of everyone’s suggestions (and some luck).
Essentially, while GPT can understand up to 4,000 words at a time, it’s really not good at understanding those words if they contain too many semi-complex to complex instructions at one time.
I broke it up into multiple functions with different user instructions I needed. Then, in the system message, I put some variation of:
“While writing, try to use as many words as you can from this list of words where they fit organically: {keywords}”
This did the trick. The biggest drawback is it takes 3-4x longer to generate. However, I’m getting ~95% word usage, even when the list is 20+ keywords, so it’s worth the trade-off for me.
Appreciate the suggestions everyone, helped a great deal.
In a discussion about forcing the GPT-3.5 Turbo API to use specific keywords in generated articles, many strategies were suggested to improve performance. User lostinsauce initially raised the issue, struggling to get the API to use a keyword list in their Python web app. The code implemented interactions between user and assistant but had no impact on keyword usage.
In response, _j suggested developing an article first before rewriting to integrate keywords organically. This suggestion stemmed from the AI’s lack of internal memory, urging that the AI should first see what it had written to guide keyword inclusion. However, lostinsauce indicated this method was unsuccessful and considered using logit bias for tokenizing keyword inputs.
_chrstfer and elmstedt suggested various methods like instructing the AI more directly, biasing certain tokens using logit bias, and adopting a more complex instruction set, even leveraging the assistant’s system message to guide actions. However, lostinsauce explained the instructions were too complex and could cause AI to timeout, especially with client-specific keywords which are non-negotiable for SEO purposes.
While there were further discussions about fitting a large list of keywords organically using several API calls, with some optimization on the importance of SEO, lostinsauce eventually found a solution via a combination of suggestions. It required breaking instructions into multiple functions, followed by a system prompt to use as many words as possible from a given list organically. This increased keyword usage substantially but extended the generation time by 3-4 times.
