How can I guarantee by prompt that ChatGPT respects words' limit?

Prompting
, ,
1

Prompteers,

I have a very simple prompt:
Summarize {{text}} in less than {{wordLimit}} words.

Unfortunately, ChatGpt (API +gpt-4 model) is not repecting the words’ limit. Even if I add extra instructions to count the number of words of the summary and compare to the limit provided, It doesn’t work all the time.

Is there an additional instruction that I can use to have 100% of accuracy?

Thanks,

Gustavo

2

The prompt that you can use, when working an API model, would be:

“You can’t count words, because you have no internal thought mechanism where the state of a growing word count can be remembered as you write, nor do you have the planning ability to produce a whole passage of a certain length, nor can you observe words well because of the tokenization encoder that encodes words into varying counts of tokens. Therefore, if the user desires an exact word count, you MUST use the provided function named count_output_words to evaluate a preliminary answer, continuing to iteratively call count_output_words function, until you have composed and discovered a composition that this tool verifies is of the correct length. Then produce the identical response for the user”

Then you must provide that code function.

All of the prompt language that justifies by explaining what the AI can’t perform is actually for your benefit, the answer to this topic.

1 Like
3

Hi @gustavomoises !

It is not actually possible for an LLM like GPT-4 to respect the word limit to within a 100% accuracy - this is mostly because LLMs work on byte-pair encoded (BPE) tokens, and not actual words. There is no prompt out there that will guarantee word limit with 100% accuracy.

There are two strategies you can employ:

  1. Make multiple “summarization” calls with the same prompt, count the words in the responses, and pick the response that meets your word limit.
  2. Make a first call to summarize within some word limit. Then count the words in the response, and if it’s above the word limit, pass this response to a 2nd call to GPT-4 and tell it to summarize. Keep doing this recursively until the word limit is met. This is iterative summarization.

So in conclusion: you have to put in bit more effort post-hoc to get this to work 100% accurately.

4

My thoughts exactly.

Or if you have access to gpt-o1 API, it would probably solve that too.