Embedding Results Scale Seems Off

[curt.kennedy]

The minimum cosine similarity of 0.7 from ada-002 is a “feature”.

You’d think it should have a min of -1, right? Well it doesn’t. The model isn’t isotropic, and has all correlations between 0.7 and 1. (Instead of ranging from -1 to 1).

You could batch process a set of embeddings, and transform them to process out this feature using PCA, but it could be more work than is necessary.

Instead, what I do, is calibrate my correlations, similar to a “0.7 -1” mapping.

Also tighten the limits of “closeness”. So instead of thinking everything within 0.1 is close, think more like 0.01 or even 0.001.

More info on solutions and motivation:

Paper here:

Some questions about text-embedding-ada-002’s embedding

OK, here is the solution. It is basically can be solved by post processing. Apparently this is a problem for trained embeddings out-of-the gate. The technical term for what ada-002 is is that it isn’t isotropic. One big vector is taking over the space and essentially reducing ada-002’s potential (dimensionality). Post-processing can improve this. Now, the paper shows the improvements are slight (2.3%), but it can be done.

And implemented it a few posts later here:

Some questions about text-embedding-ada-002’s embedding

Hey @ruby_coder @debreuil Here is the code I wrote to do this. Hope it helps. import numpy as np import sklearn.decomposition import pickle import time # Apply 'Algorithm 1' to the ada-002 embeddings to make them isotropic, taken from the paper: # ALL-BUT-THE-TOP: SIMPLE AND EFFECTIVE POST- PROCESSING FOR WORD REPRESENTATIONS # Jiaqi Mu, Pramod Viswanath # This uses Principal Component Analysis (PCA) to 'evenly distribute' the embedding vectors (make them isotropic) # For more information o…

After post processing ~60k embeddings, I was finally getting 0 and negative dot products. The values “made sense” empirically, but there was no way for me to fully validate all the results. But give it a shot, especially when you care about geometry, to do, for example, analogy searches (on word or sentence level), like so: