Is it possible to build contextualized embedding with GPT?

I have used “text-embedding-ada-002” to build embeddings from texts, however I would like to know if I can build better embeddings by giving GPT, maybe gpt 3.5 turbo, the context from my texts.

This idea came from another embedding model called INSTRUCTOR, that can be found in this link GitHub - HKUNLP/instructor-embedding: One Embedder, Any Task: Instruction-Finetuned Text Embeddings, and it builds embeddings following certain instructions, like: "Represent the Medicine sentence for clustering: "

I believe that using that I could improve my embeddings considerably for future clusterization, searching or other tasks.

1 Like

It is a good topic.

I also find that generative pre-training can bring some text similarity.

What is the situation of your research now?