Any API end point to display token usage for the query

API
1

I am working on a tool to make API call to display response on the wordpress post. I want to know if there is any API endpoint which is available (token counter?) which I can use in my script show that it displays number of tokens used in that query on the same page.

Also, is there any option that I put max tokens usage for each API key so as soon as it hits that cap, it stops working so that I can ask my customers to upgrade to a different or higher plan. Thanks

1 Like
2

Yeah the current API displays the total of tokens used in the returned response.

"usage": {"prompt_tokens": 23, "completion_tokens": 1, "total_tokens": 24}}

As far as I can tell, it is across fine-tuned models and normal models as well.

2 Likes
3

That is fantastic. I will definitely try to use these endpoints and see how I go. Thanks a lot.

Still looking for a second question… is there a way to put a cap on each API for my customers?

Thanks

4

Yes, you can cap it for each user when you send the request. Set max_tokens to the max allowable total tokens, which is the sum of input + output.

5

Thank you so much. Sorry if I didn’t specify my question properly. I wanted to know if there is anything I can do so that a particular API stop working as soon as they use their total tokens (whatever I allocate them when they sign up on my website). For example, if I have allocated 20K tokens a particular API key, as soon as that key consumes that total token (e.g. by searching for multiple prompts in a week or so), they get an error that you have consumed your monthly quota and you need to upgrade your account to continue using the tool. Thanks

6

That is something you will have to track with your own database. Not aware of any such tracking built-in to the current API.

1 Like