Questions on the use of text-embedding-ada-002 model

[tzortzinak95]

Hello everyone,

I am quite new to text embeddings and text comparison in general, but I want to use text-embedding-ada-002 to compare a job description with various resumes. The idea is to sort the resumes based on their similarity with the job description.

However, I have a little difficulty in understanding the results I am getting.

The process I am following is the below:

1. Extract the text from the job description pdf
2. Clean the text by removing non ascii characters, punctuation, new lines and converting to lowercase
   3)Tokenize the text with the below code

enc = tiktoken.encoding_for_model("text-embedding-ada-002")
tokenized_jd = enc.encode(jd)

4. Get the embedding with the below code
   job_description_embedding = openai.Embedding.create(encoding_format="float", model="text-embedding-ada-002", input=[tokenized_jd])['data'][0]['embedding']

5. For each resume repeat the same process, get the embedding and then compare the job description embedding and the current resume embedding by using
   cosine_similarity(job_description_embedding, candidate_resume_embedding)

What is puzzling to me is that all the comparisons show very high similarity even if the resume has nothing to do with the job. For example, the lowest similarity score I have gotten is 0.72 for a resume that from a nutritionist even though the job description was for a senior java developer so I was expecting a lower similarity score.

I wonder if anyone can shed some lights on this? Is this normal? Am I using ada002 wrongly or could there be another reason for these results?

Thank you in advance

[curt.kennedy]

This has been discussed extensively before, basically the embedding vectors from ada-002 get squished together, leading to high correlations no matter what. You just need to adjust your correlation expectations, or you can batch process out the vector correlation using PCA to make them more isotropic (spread out) in the future.

See …

Some questions about text-embedding-ada-002’s embedding

Hey @ruby_coder @debreuil Here is the code I wrote to do this. Hope it helps. import numpy as np import sklearn.decomposition import pickle import time # Apply 'Algorithm 1' to the ada-002 embeddings to make them isotropic, taken from the paper: # ALL-BUT-THE-TOP: SIMPLE AND EFFECTIVE POST- PROCESSING FOR WORD REPRESENTATIONS # Jiaqi Mu, Pramod Viswanath # This uses Principal Component Analysis (PCA) to 'evenly distribute' the embedding vectors (make them isotropic) # For more information o…

Expected Angular Differences in Embedding Random Text?

I have a dataset with over 80k random text messages and I embedded each of the messages with ‘text-embedding-ada-002’. When I pick a message at random, and find the top 10 messages close (+1 dot prodoct), far away (-1 dot product) and orthogonal (0 dot product), all I get are embeddings that are at most 50 degrees away! The messages range over random spammers and alerts to more common messaging you would expect from millions of people. So I expect to see embeddings that at least have a negati…

[CreatiCode]

@tzortzinak95

the lowest similarity score I have gotten is 0.72 for a resume that from a nutritionist even though the job description was for a senior java developer so I was expecting a lower similarity score.

The similarity score is more useful for comparing with each other. If you choose another resume about a developer, do you get a higher similarity score?

[tzortzinak95]

Thank you very much for sharing! I will definitely try it out!

[tzortzinak95]

Yes most of the time in this case the similarity scores I have gotten are quite higher. Mostly around 0.8 to almost 0.9