I’m currently using separate API keys for my production and staging environments. I’d like to clarify how TPM (tokens per minute) limits are applied. Are TPM limits accounted for independently on a per-API-key basis, or are they shared across all keys under the same account?
In other words, if my staging environment hits its TPM limit, could this also impact my production environment’s access (even though it uses a different API key)? Any insights or guidance from those who have encountered a similar situation would be greatly appreciated.
Overall rate limits are on the organization level.
Within individual projects, you can place further rate limits yourself. You might keep an app developer from going to $1000 per hour when five requests per minute are adequate.
There are different groups of rate limits by model. For example, embeddings is a different pool than any completion model. You can look at your rate limit page in the platform account, and infer the shared rate pools of those models that have the same limits.
Hitting a rate limit will produce an API error. If it is the organization limit for a model class, everyone will get the error until the rate algorithm “refills”.
From your explanation, I understand that the TPM and related rate limits are indeed applied at the organization level, not just per API key.
This means that if our staging environment hits a rate limit, it will affect production as well.
We’ll keep this in mind and plan our usage accordingly.
Thanks again for your help!
Another strategy for true isolation is to use a second organization.
“Organization” by name is something that is contained within accounts, but can also be shared with other accounts, or another account can even be made “owner” instead of “reader” for full management (and even takeover).
Each organization has its own billing, and must individually reach payment tier qualifications to have higher usage limits.
Some older accounts had a second “personal” organization added that could be employed for this purpose. At one point in time, you could also request an organization be added by making a request to OpenAI, but I don’t know the current policy. If another account was made owner and took your organization, removing you, then there would be no organization within the original account for API, necessitating requesting a new organization.