Evaluations Beta custom eval prompt

I’m trying to evaluate my stored prompts with the new Evaluations feature. I want to use custom evaluations via a prompt. Most of the times all evaluations fail. Is there a way to see the evaluation response to check what’s going on? I think the issue is most likely related to the evals prompt not outputting the exact label.

Also, it states that you can use {{}} for placeholders. But what placeholders are there? (input & output?)

The current docs about this are pretty limited…

Thanks a ton :muscle:t3:

2 Likes

It seems that the current state of the Evaluations feature is mega-beta. I can’t even create one (cannot add testing criterias)