We have a multi-tenant architecture with all tenants using our OpenAI API key. We want to track LLM costs per customer (and for every feature they use). The usage dashboard provided by OpenAI doesnt work because we use the same key for all customers. Is there a way for us to breakdown the usage per customer? Maybe there is a way for us to provide additional meta data while calling the OpsnAI APIs. Or the other way is for us to ask customers to use their API keys but then we lose the analytics of which AI feature is being used the most. For now we are logging customer_id, input_tokens, output_tokens for every API call. But wondering if there is a better solution here.
Have you considered creating a project per tenant? Can be done in the API and you can then use reporting per project (also through API)
1 Like
Dont know if that would scale when my customer base grows as it will be hard to manage 1000s of individual projects (though a good problem to have). Also, ideally, I would want to track which features are being used by my customers. For example, feature A (which uses OpenAI APIs) is the top feature used by customer X, and feature B (which uses OpenAI APIs) is the top feature used by customer Y. But I can try using projects to start with.
A solid approach is to use a gateway to handle all requests to and from the model. This makes it easier to track feature usage and costs per user.
That said, if projects meet your needs, they’re a great alternative.
This is a good idea. We are kinda doing this way right now. We use one central API client that routes all the requests to OpenAI and also logs to S3 for our analysis later. We log things like customer_id, feature_id, input_tokens, output_tokens, model, cost etc. Then we can use Athena to do some analytics. But I feel this is super basic so was looking for more robust way.
1 Like
What exactly are the additional requirements to satisfy your needs for additional scaling?
As I mentioned, being able to track the API usage at each feature level. Just as we are able to all the analytics for a website and able to find out which feature is being and which one is not.