# Evaluating LLM Chat Responses without Evaluation Dataset?

**URL:** <https://community.openai.com/t/evaluating-llm-chat-responses-without-evaluation-dataset/820123>\
**Category:** API\
**Tags:** gpt-4, assistants-api\
**Created:** [June 14, 2024, 6:56pm UTC](https://community.openai.com/t/evaluating-llm-chat-responses-without-evaluation-dataset/820123 "2024-06-14T18:56:51Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![johnald](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/johnald/32/149204_2.png) [@johnald](https://community.openai.com/u/johnald)\
**Post date:** [June 14, 2024, 6:56pm UTC](https://community.openai.com/t/evaluating-llm-chat-responses-without-evaluation-dataset/820123/1 "2024-06-14T18:56:51Z")

</div>

Hey all,

Was wondering what the best way of evaluating LLM chat responses were without an evaluation dataset with “expected” outputs. Chat responses are non-deterministic and therefore don’t have fixed/predictable outputs, and benchmarks such as MMLU/HELM often can’t distinguish how well a chatbot is performing, as noted in the LLM-as-a-Judge paper.

Would greatly appreciate any insights on how people evaluate these LLM responses without a defined eval test suite, thanks!

---

<div class="post-metadata">

**Author:** ![anon10827405](https://avatars.discourse-cdn.com/v4/letter/a/ed8c4c/32.png) [@anon10827405](https://community.openai.com/u/anon10827405)\
**Post date:** [June 14, 2024, 7:46pm UTC](https://community.openai.com/t/evaluating-llm-chat-responses-without-evaluation-dataset/820123/2 "2024-06-14T19:46:57Z")

</div>

What do you expect is really what it boils down to.

You can easily give the essence of the “answer” to an LLM for grading. It doesn’t need to be exact. All that matters is that the semantics _somewhat_ resemble what’s expected & capture what you’re expecting

---

<div class="post-metadata">

**Author:** ![ra\_0929](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/ra_0929/32/284555_2.png) [@ra\_0929](https://community.openai.com/u/ra_0929)\
**Post date:** [June 14, 2024, 7:49pm UTC](https://community.openai.com/t/evaluating-llm-chat-responses-without-evaluation-dataset/820123/3 "2024-06-14T19:49:03Z")

</div>

The only way I could think of was random input. In keeping with your context.  
I just query mine with the most obtuse things I can think of. And note the responses. But there are random word generators that can pipe to your prompt. For instance Crunch found at [crunch - wordlist generator download | SourceForge.net](https://sourceforge.net/projects/crunch-wordlist/)