How to get token usage for each API call in streaming model?

klcogluberk · 2023-04-24T07:32:07.416Z

The response like this:

{
  "choices": [
    {
      "delta": {
        "content": "Hab"
      },
      "finish_reason": null,
      "index": 0
    }
  ],
  "created": 1680676704,
  "id": "chatcmpl-.....",
  "model": "gpt-3.5-turbo-0301",
  "object": "chat.completion.chunk"
}
{
  "choices": [
    {
      "delta": {},
      "finish_reason": "stop",
      "index": 0
    }
  ],
  "created": 1680676704,
  "id": "chatcmpl-.....",
  "model": "gpt-3.5-turbo-0301",
  "object": "chat.completion.chunk"
}

when I send an API call with stream=False, token_usage information was sent in response body. But when i set stream=True, there is nothing like this in the response. How can I calculate the amount of tokens used by myself.

sajo · 2023-05-03T06:12:18.245Z

I have the same request. Please provide token usage in stream mode like you provide “finish_reason” in the last chunk.

PaulBellow · 2023-05-03T06:18:22.449Z

Genuinely trying to be helpful here…

helpful587×755 26.8 KB

How to determine the token usage for a session when using stream:true? API

I am using the Node.js API. Previously, when stream:false, the API would return prompt_token and completion_token (maybe these are the field names). But after using streams, I cannot find these two fields. How can I accurately retrieve them? --Translated from ChatGPT

OpenAi API - get usage tokens in response when set stream=True API

However, after extensive testing, I found that the token value calculated by the calculator for offline token calculation is far from the actual value used. So Python’s tiktoken is not reliable. However, please find attached the method I am currently using to calculate tokens: Each stream containing an answer is treated as a token, and when all these streams are added up, they are equal to all the tokens in this question. This is the method for calculating the response token. The token method…

hemanthmurugan21 · 2023-06-06T07:18:32.562Z

use tiktoken GitHub - openai/tiktoken: tiktoken is a fast BPE tokeniser for use with OpenAI's models. this link .cumulate all tokens and calculate

github.com

openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to count tokens with tiktoken\n",
    "\n",
    "[`tiktoken`](https://github.com/openai/tiktoken/blob/main/README.md) is a fast open-source tokenizer by OpenAI.\n",
    "\n",
    "Given a text string (e.g., `\"tiktoken is great!\"`) and an encoding (e.g., `\"cl100k_base\"`), a tokenizer can split the text string into a list of tokens (e.g., `[\"t\", \"ik\", \"token\", \" is\", \" great\", \"!\"]`).\n",
    "\n",
    "Splitting text strings into tokens is useful because GPT models see text in the form of tokens. Knowing how many tokens are in a text string can tell you (a) whether the string is too long for a text model to process and (b) how much an OpenAI API call costs (as usage is priced by token).\n",
    "\n",
    "\n",
    "## Encodings\n",
    "\n",
    "Encodings specify how text is converted into tokens. Different models use different encodings.\n",
    "\n",

This file has been truncated. show original

wolf · 2023-07-04T17:12:52.959Z

I use the API from Javascript not Python. So how can I calculate the request tokens.

By the way. In my oppinion the missing token-information it is an open issue.

elmstedt · 2023-07-04T17:17:15.166Z

Perhaps this?

GitHub

GitHub - niieani/gpt-tokenizer: JavaScript BPE Tokenizer Encoder Decoder for...

JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features. - GitHub - niieani/gpt-tokenizer: JavaScript BPE Tokenizer E...

wolf · 2023-07-04T18:28:23.557Z

Looks good, but when I include it into my Page

const { encode, decode, encodeChat } = GPTTokenizer_cl100k_base
const chatTokens = encodeChat(chat,'gpt-3.5-turbo')

I get the following error:

gpt-tokenizer:1 Uncaught TypeError: Cannot read properties of undefined (reading 'encodeChatGenerator')
    at encodeChat (gpt-tokenizer:1:2093142)
    at Object.javascriptFunction (home?session=8369786809309:842:20)
    at da.doAction (desktop_all.min.js?v=22.2.4:24:5629)

so can you help me how to use it?

elmstedt · 2023-07-04T19:08:11.392Z

I’m sorry, I don’t have any personal experience using that package.

I’m a bit busy today, but there’s a chance I can take a look at it tomorrow if you’re still having issues.

wb · 2023-07-06T08:04:22.195Z

Hi, did you find a way of using it in a normal website?