In playground when using Chat mode, there’s this section named System, where you can put instructions on what type of output I want through an example. do these tokens get charged for each user message submitted. Like are these tokens submitted each along with my user message?
Hi and welcome to the developer forum!
Yes, all text sent to the model is counted towards tokens used irrespective of their location in the playground layout.
2 Likes
Is there any way to prevent this. I know finetuning but I don’t have a large enough dataset to do so.
You might look at embeddings, but the payoff ratio will be on a case by case basis as to the effectiveness of including an embedded context retrieval segment with each prompt, compared to simply including the full text with each prompt.
If you have many pages of documentation that needs to be utilised then embeddings can be of use, if it is a few lines of text, then it won’t
2 Likes
No, you cannot avoid this.
Fine-tuning isn’t a solution if your goal is to reduce cost—the cheapest fine-tuned model ada cost more to run than gpt-3.5-turbo without even counting the training costs, plus the results will likely be much worse.
If the cost of including a system message is make-or-break for you, then I’d suggest downloading and running llama-7b instead.
1 Like