Exploring AI's Capability to Evaluate Conversations: A Feasibility Inquiry

Hello everyone,

I’ve been deeply intrigued by the intersection of artificial intelligence and human communication. I wonder about the possibility of creating a system that can evaluate conversations in various aspects (like coherence, depth, empathy, creativity, etc.) and assign a numerical value from 0 to 10 based on its analysis. This approach could offer new insights into the quality of our interactions, serving as a valuable tool for both personal development and communication research.

Do you know if there IS any prompt approach that could fit with this task??

1 Like

I think most of the aspects you mention can be subjective in nature

Why not ask the end user to rank their conversation?

Alternatively, using prompts may not be enough. You will need to train a model so that it is able to identify the various aspects. But then, again, the subjectivity of the one training the model may come into play, no?

For example, in some of my API conversations, some users found the bot to be very “friendly”, whereas others found it “shallow” or “callous” - for exactly the same kind of responses for similar questions!

Human subjectivity is complex

ok, but i dont want that the end user rank the conversation, i want to rank different end users that had a conversation with the same assistant.

At the end, all of them are speaking assistant prompt, and with the same model, so based on their conversations should be possible rank them automatically in different aspects, dont judging the AI in this case.

For sure, after you have the first evaluations for each conversation maybe its necessary to validate the punctuations that the second Assistant assign to each conversation, but if U use a good prompt there should be some coherence between the assigned punctuation.