Question on text-embedding-ada-002

[curt.kennedy]

@jz97

In the last example, where the sentence is actually “long” but the ONLY meaningful distinction is the word “positive” vs “negative”. the similarity is at a whopping 0.942!

I wonder if anyone can shed some lights on this?? are there any known limitation on ADA002? or maybe i am using ada002 wrongly?

Yes, ada-002 has poor geometry and is non-isotropic.

I mentioned it initially a while ago over here:

Expected Angular Differences in Embedding Random Text?

I have a dataset with over 80k random text messages and I embedded each of the messages with ‘text-embedding-ada-002’. When I pick a message at random, and find the top 10 messages close (+1 dot prodoct), far away (-1 dot product) and orthogonal (0 dot product), all I get are embeddings that are at most 50 degrees away! The messages range over random spammers and alerts to more common messaging you would expect from millions of people. So I expect to see embeddings that at least have a negati…

And I came up with a solution that involves PCA (Principal Component Analysis) fitting of the data over here:

Some questions about text-embedding-ada-002’s embedding

Hey @ruby_coder @debreuil Here is the code I wrote to do this. Hope it helps. import numpy as np import sklearn.decomposition import pickle import time # Apply 'Algorithm 1' to the ada-002 embeddings to make them isotropic, taken from the paper: # ALL-BUT-THE-TOP: SIMPLE AND EFFECTIVE POST- PROCESSING FOR WORD REPRESENTATIONS # Jiaqi Mu, Pramod Viswanath # This uses Principal Component Analysis (PCA) to 'evenly distribute' the embedding vectors (make them isotropic) # For more information o…

Realize that ada-002 only requires dot-products as a distance metric (equivalent to cosine similarity since ada-002 consists of unit vectors.). If you had a TON of embeddings and needed a speedup in your search, you could drop down to a Manhattan metric instead, since this only involves additions (subtractions) and absolute values.

But with a 400k set of embeddings using exhaustive search only takes 1 second using dot-products (inner-products) and basic cloud functions.