De-biasing of ada embeddings

[curt.kennedy]

There is a large vector bias and vector correlation with ada-002 embeddings.

You could increase your thresholds, or if you have a batch of embeddings, you could remove the bias and use PCA to spread them out, like I did here:

Some questions about text-embedding-ada-002’s embedding

Hey @ruby_coder @debreuil Here is the code I wrote to do this. Hope it helps. import numpy as np import sklearn.decomposition import pickle import time # Apply 'Algorithm 1' to the ada-002 embeddings to make them isotropic, taken from the paper: # ALL-BUT-THE-TOP: SIMPLE AND EFFECTIVE POST- PROCESSING FOR WORD REPRESENTATIONS # Jiaqi Mu, Pramod Viswanath # This uses Principal Component Analysis (PCA) to 'evenly distribute' the embedding vectors (make them isotropic) # For more information o…