How to save tokens with the Instruct series

Hey everyone!

Recently I found a little trick you can use to minimise the amount of training data required but still get the results you’re looking for. This is an example case, but applies to much more complex prompts aswell.

The Idea
The idea is that you can supply training data by clearing specifying the number of data points your supplying. When you go to query for a completion, simply specify the larger number of results you’re looking for (6 in the case of this example) and it will return that quantity.

image

Why is this important?
I found myself supplying for example 6 training samples in order to get 6 back. The ability to then reduce that by 60ish% is a huge save on tokens when prompts are 500-700 tokens.

Note
This only works on the instruct series (as per my testing) especially with more complex prompts.

Hope this helps somebody save tokens!

5 Likes

Interesting, I was able to get the desired result with curie-instruct and it seemed to work flawlessly.

image

If the issue is a stop token, then I guess adding two newlines to your stop sequences is equivalent.

1 Like