Use cli to batch download the usage data of openai api

Community
1

A CLI tool for downloading data from the OpenAI API to facilitate further analysis of usage.

Usage

Prerequisite: Install nodejs 18 lts


npx @ai-assist/openai-apilyzer --authorization 'Bearer <access_token>' --organization 'org-<organization>'

# downloaded
# written to openai-api-usage-2023-02-28-2023-03-30.csv

options source

image
image2344Ă—1706 260 KB

The CSV file can then be viewed in Excel.

image
image1920Ă—1191 288 KB

Options

npx @ai-assist/openai-apilyzer -h
Usage: openai-apilyzer [options]

Options:
  --authorization <authorization>  authorization token
  --organization <organization>    organization id
  --start [start]                  start date, default is 30 days ago (default: "2023-03-14")
  --end [end]                      end date, default is today (default: "2023-04-13")
  --format [format]                json/csv format of output, default is json (default: "csv")
  --output [output]                output file, default is openai-api-usage-{options.start}-{options.end}.{options.format}
  --baseUrl [baseUrl]              base url of openai api (default: "https://api.openai.com")
  --verbose                        verbose output (default: false)
  --userPublicId [userPublicId]    user public id
  -V, --version                    output the version number
  -h, --help                       display help for command

Note that if you are in China or other regions where access to the openai API is not possible, you can use the --baseUrl parameter to specify a mirror address, such as https://openai.ai-assist.moe, which can be deployed to a Cloudflare Worker at zero cost through the open source project openai-proxy.

This is an idea I came up with while dealing with another problem, ref: What is n_context_tokens_total? Why is it so large?

1 Like
How to track individual usage?
2 
start downloading
downloading 2023-03-14
Error occurred (fetch failed). Retrying (2 attempts left)...
Error occurred (fetch failed). Retrying (1 attempts left)...
Error occurred (fetch failed). Retrying (0 attempts left)...
download failed, please check your authorization and organization id

My node.js fetch code can’t work either. But if I copy curl from chrome, it works.
Is this tool still working or openai has raised protection for API endpoints?

3

Release 0.1.9 fixed, it seems that openai modified the http header part, I made some adjustments, it works now. Another reminder, if you are in an area where you cannot access the openai api, please configure the proxy correctly, or use a mirror service, and refer to the updated baseUrl field.

4

I got same result @0.1.9. I ran it on a server located in the US.

5

Can you try 0.1.10 and add --verbose option? It will print out verbose error log

7

It works! Thanks!
Now I’m trying to implement count usage by individuals.

8

Release 0.1.13, already supports parameter --userPublicId

9

In the end, I created a chrome extension to export it, so I didn’t need to install any runtime locally, and I didn’t need the command line. The chrome store is currently under review, and the zip of the plugin has been built and can be installed locally to try.

image
image334Ă—514 13.6 KB

10

Approved by the chrome app store

11

Does this still work? I’ve been unable to get it to work.

12

No, I’m sorry, but I no longer maintain it.

13

OpenAI has locked down the usage API that the platform site uses pretty hard against bots. What was an “undocumented API” is now a “good luck” API.

However, you can look at the administration API surface, needing generation of an admin key. See if the resources you want are right there.

https://platform.openai.com/docs/api-reference/usage/completions

14

Thanks, Jay. I’ll try that.