Pricing details RE: Evals feature

API Feedback
, , , ,
1

Hello!
I have been searching in vain for more details concerning the pricing of the Evals tooling – either via API/github repo, or via the playground. I understand that any tokens consumed are priced at the “regular” rate (although the playground doesn’t specify which model you are using) but it is not clear to me whether there is an additional charge to run a single instance of an eval on top of the cost of token usage.

This is an important variable cost metric for our startup business, as we may look at “rolling our own” by leveraging RAGAS or something if it proves too cost prohibitive to call OpenAI Evals at scale. More importantly, I would like to avoid a scenario in which we build a scalable OpenAI Evals integration and find a bunch of unforeseen charges beyond the token usage cost.

Please lmk! Am impressed by the tooling so far, but the economics need to make sense for us.

Best,
Zack

2

Have you found out the pricing? I am in a similar situation.

3

Hi!
The only costs are the token costs. All the other features (report page, api, etc.) are not charged.

Only token costs for generating responses and if you use the model graders.

2 Likes
4

Rather: the program for “7 free” ended last year. There is no complementary consideration for being submitted through evals.

I don’t expect that “complementary tokens with training on your prompts” would be affected, but me being wrong about the eval’s method of backend model usage could mean being billed those also.

Other works by the author.