What is the method used behind cosine similarity in the evaluation?

Community
1

Hello,
While building evaluation pipeline on the dashboard, I came across various textual similarity metrics, one of them being cosine similarity. Is there any information on the method used to compute the vectors for the cosine similarity calculation? Is an embedding model being used?
Thank you.

1 Like
2

Hi,

Welcome to the Developer forum.

I believe this is cosine similarity

(You can probably find a more appropriate video on YouTube)

https://platform.openai.com/docs/guides/embeddings

The image shows a table comparing the cost per million tokens for different embedding models, with prices of $0.02 for text-embedding-3-small, $0.13 for text-embedding-3-large, and $0.10 for text-embedding-ada-002. (Captioned by AI)
The image shows a table comparing the cost per million tokens for different embedding models, with prices of $0.02 for text-embedding-3-small, $0.13 for text-embedding-3-large, and $0.10 for text-embedding-ada-002. (Captioned by AI)1167×308 18.7 KB

https://platform.openai.com/docs/pricing

1 Like
3

In the evals API endpoint and its platform UI, this is what is being discussed.

The image displays a grid of icons with descriptions for different testing criteria such as factuality, semantic similarity, sentiment, and more. (Captioned by AI)
The image displays a grid of icons with descriptions for different testing criteria such as factuality, semantic similarity, sentiment, and more. (Captioned by AI)811×888 102 KB

One can also visit the API Reference and see the actual graders, which have little correlation to the UI.

The image shows a detailed description of the ScoreModelGrader object, which assigns a score to input text using a specified model. (Captioned by AI)
The image shows a detailed description of the ScoreModelGrader object, which assigns a score to input text using a specified model. (Captioned by AI)523×782 14.9 KB

This post highlights a slight complete absence of any useful documentation at all.

One expect that “cosine similarity”, analogous to a normalized dot product, (wherever you might be encountering that), refers to the typical algorithm for comparing semantic similarity between two embeddings vectors.