Thats great if its appropriate for your applications.
Most of the projects we’ve worked on are mobile and distributed.
They also use the streamed response mechanism that doesnt provide the cost information.
Clients who use the applications are also not interested in having their requests go through some API gateway I operate.
The OpenAI system lets developers maintain multiple keys. We can even name them, and see when they were last used. It seems like a reasonable request to be able to see the accounting for how much each key was used in a billing cycle.
Personally, I would rather handle the billing / usage on my end and let them concentrate on bigger / better models rather than trying to build out infrastructure for payments. That said, we might see it in the future? Although I bet it would show up on Azure first…
It’s obviously possible to track this outside of openAI, but it’s clumsy and always includes some degree of inference. Should be available at the source - openAI Usage screen.
The discussion focuses on tracking usage and cost by individual API keys in relation to OpenAI. Users bzhang, carausack, omk, and others are seeking a solution to view data by API keys, which seems unsupported since usage tracking is by organization.
jwatte suggests a workaround using a separate logging system for each API request and then summarizing the data. The user also mentions Observe Inc as a solution, but notes it necessitates extra instrumentation and query setup. nguyenanhdon.qn shares a command to view API usage for a specified date range.
SomebodySysop indicates they are manually logging token usage with other parameters like user, model, date, and IP address. juan_olano details a method that involves associating each API request to a userID, capturing the returned token usage and associating it with the userID in a database.
Several users such as AI-Roguelite and _j suggest creating separate accounts for each API key or using organization-based tracking solution as workarounds. _j provides a link to a GitHub script showing an endpoint for tracking organization user usage.
will.hawkins00 shared that using the GET command was met with an error requiring a “session key”. _j responds suggesting usage of the HTTPs TSL session key, only available when authenticated on the OpenAI website.
jochenschultz suggests using an API gateway for centralization, logging and control over key usage, along with a queuing system to avoid exceeding maximum tokens per minute. Robertsmania points out that this solution may not always be practical, especially for mobile and distributed projects. Several users including happy2chat, davedevs, and badcom showed support for OpenAI to introduce this feature in the future.
Summarized with AI on Nov 28 2023
AI used: gpt-4-32k
As I know, there is no official API. What I’m doing is use Cloudflare to proxy OpenAI requests, and provide self-distributed api key for different purpose and count the tokens. Though it may not be that precise, but it works perfect.
You can build a simple LLM API gatway using fastapi with a user auth flow to generate API keys and then use those to monitor costs per user (and control them through checks in your API gateway on every user call)
New ways to understand API usage and manage API keys
We are launching two platform improvements to give developers both more visibility into their usage and control over API keys.
First, developers can now assign permissions to API keys from the API keys page. For example, a key could be assigned read-only access to power an internal tracking dashboard, or restricted to only access certain endpoints.
Second, the usage dashboard and usage export function now expose metrics on an API key level after turning on tracking. This makes it simple to view usage on a per feature, team, product, or project level, simply by having separate API keys for each.
the link to “turning on tracking” leads to the page for creating a new api key. neither that page nor the dialogs for actually creating an api key have any obvious way to turn on tracking. is it automatic for all new api keys?
