Poor embedding performance using ada for portuguse

[christiancleber]

I created two embeddings using ADA for two words in Portuguese: cimento (cement) and sorvete (ice cream). The cosine similarity between them was 0.8 what is clearly wrong. Any thoughts?

[_j]

For single word embeddings in a distinct language, I think you picked the wrong model. ada is best for comparing larger texts and user inputs to texts.

Repositório de Word Embeddings do NILC

The best performance for semantic word embeddings there is reported as glove 600. You can see how to employ the Gensim model and obtain single word vectors here:

Word2vec Tutorial | RARE Technologies

[christiancleber]

Just tested for larger texts and the results are still bad.

Using - stjiris/bert-large-portuguese-cased-legal-tsdae-gpl-nli-sts-MetaKD-v0 which is a Sentence Transformer for Portuguese, the results are consistent for single words or sentences.

I would expect that OpenAI Sentence Transformer would be far better than anyone else.

[curt.kennedy]

Even in English, you will get a cosine similarity of 0.8 for non-related things.

I have talked about this problem to death. So there are two solutions.

1. Easy: Tighten your bounds for similarity.
2. Hard: Post process the embedding vectors, removing the correlation and biases in the model that are causing this.

This is not a Portuguese thing! It’s an Ada-002 thing!

[christiancleber]

Yes, that’s exactly what I got. 0.8 for any kind of comparison.

Thanks for clarifying!

I see a lot of tutorials teaching how to do RAG (with OpenAI) and the basic principle is to search a vector database to filter relevant text associated to the question. This search is made via semantic similarity. How this can work reasonably well given this cosine similarity behavior.

[curt.kennedy]

@christiancleber

How this can work reasonably well given this cosine similarity behavior.

It works for most because folks aren’t paying attention to the correlation value. They are only grabbing the top K highest scoring things. So the non-isotropic behavior of the model is swept under the rug.

They aren’t looking for de-correlated things, or orthogonal things. Or measuring how uncorrelated your “top K” results really are. Another reason is that, with RAG, they feed the top K answers into the LLM, and let the LLM decide if it’s related or not.

So because the LLM can also sort out the non-correlated results, it also sweeps the issue under the rug.

There are many theories on why this is happening in the model, like overcompensating for hidden states. And it seems to happen in most models, but it is really bad in ada-002.

Here is the paper discussing this:

[Paper] AstroLLaMA: Towards Specialized Foundation Models in Astronomy

Sorry meant to say non-isotropic, with the ‘r’. Here us a paper for more context. Previously I have used ABTT, mentioned in the paper, to help resolve this.

Also, here is code that I implemented “ABTT” that essentially de-biases and de-correlates your embeddings.

Some questions about text-embedding-ada-002’s embedding

Hey @ruby_coder @debreuil Here is the code I wrote to do this. Hope it helps. import numpy as np import sklearn.decomposition import pickle import time # Apply 'Algorithm 1' to the ada-002 embeddings to make them isotropic, taken from the paper: # ALL-BUT-THE-TOP: SIMPLE AND EFFECTIVE POST- PROCESSING FOR WORD REPRESENTATIONS # Jiaqi Mu, Pramod Viswanath # This uses Principal Component Analysis (PCA) to 'evenly distribute' the embedding vectors (make them isotropic) # For more information o…

[christiancleber]

Thanks a lot Curt.

I will definitely take a look on those papers.