# Converting embedding vector to text **URL:** \ **Category:** API\ **Tags:** embeddings\ **Created:** [August 29, 2023, 2:49am UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783 "2023-08-29T02:49:01Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![zhan1130](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/zhan1130/32/32807_2.png) [@zhan1130](https://community.openai.com/u/zhan1130)\ **Post date:** [August 29, 2023, 2:49am UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/1 "2023-08-29T02:49:01Z")
Could anyone explain how OpenAI embedding works? As in how are a stream of tokens transferred into one single vector? And more importantly, how to convert this vector back into the original text? I know this is in principle doable because there are literatures on gradient-based adversarial attacks (e.g., [https://aclanthology.org/2021.emnlp-main.464.pdf](https://aclanthology.org/2021.emnlp-main.464.pdf)) on nlp tasks but I have not found a clear explanation yet. ---
**Author:** ![\_j](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/_j/32/766292_2.png) [@\_j](https://community.openai.com/u/_j)\ **Post date:** [August 29, 2023, 4:52am UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/2 "2023-08-29T04:52:04Z")
Embeddings: send text. Receive multi-dimensional vector. Compare vector with prior vectors and algorithm (cosine or dot product) to get score of how similar they are. Similarity is based on AI understanding of the language using AI training, in many internal ways hard to describe. They cannot be reversed into the original language. Even trying to determine what one value of 1536 represents would take fuzzing immeasurable texts to find optimum activations or negations. * * * Researching AI internals on a model hundreds of times smaller: [Language models can explain neurons in language models](https://openai.com/research/language-models-can-explain-neurons-in-language-models) OpenAI microscope, to research what is represented in image AI: [Zoom In: An Introduction to Circuits](https://distill.pub/2020/circuits/zoom-in/) ---
**Author:** ![zhan1130](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/zhan1130/32/32807_2.png) [@zhan1130](https://community.openai.com/u/zhan1130)\ **Post date:** [August 29, 2023, 5:04am UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/3 "2023-08-29T05:04:43Z")
I see. But intuitively I think it is possible to convert the embedding vectors for individual tokens back to tokens (i.e., before positional embedding)? ---
**Author:** ![\_j](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/_j/32/766292_2.png) [@\_j](https://community.openai.com/u/_j)\ **Post date:** [August 29, 2023, 5:20am UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/4 "2023-08-29T05:20:51Z")
You can think what you want. There isn’t “positional embeddings”. The whole text is considered as a single language entity and the internal state it puts the AI in, and a single vector is returned. You could send all 100,000 BPE tokens for the trained language model individually. Each would have a vector return that is unique. Individual tokens could be reversed using that dictionary based on the same model, but 100,000^2 trials for merely two token sequences? Many single token vector elements found similar would have unobvious meaning. “aub” similar to “egg” and “veget”? Because aubergine? Vectors aren’t exposed. they are used internally. Do you have some ultimate question? ---
**Author:** ![zhan1130](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/zhan1130/32/32807_2.png) [@zhan1130](https://community.openai.com/u/zhan1130)\ **Post date:** [August 29, 2023, 6:04am UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/5 "2023-08-29T06:04:05Z")
In my last reply, I did not imply I wanted to find the original sentence anymore. In the adversarial attack example, I believe they also did it token-wise. My goal is to generate tokens that follow a specific distribution of my interest in embedding space. ---
**Author:** ![\_j](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/_j/32/766292_2.png) [@\_j](https://community.openai.com/u/_j)\ **Post date:** [August 29, 2023, 6:27am UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/6 "2023-08-29T06:27:03Z")
The adversarial “attack” doesn’t seem to have usefulness or merit, nonsense like “we assume the attacker has access to the model’s training data”. Input token modification is exactly what a user of a LLM is supposed to be able to do, and they get the results they deserve, whether from a language model or its embeddings engine that may serve internal tasks: Tell me a story please Tell me a story a\*\*hole zhan has a beautiful wife zhan has a murderous wife The “results” shown as example are changing one part of output that can be simply the non-deterministic nature of LLM. “generate tokens in embedding space”? Are you currently trying to break entailment? ---
**Author:** ![VeitB](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/veitb/32/712987_2.png) [@VeitB](https://community.openai.com/u/VeitB)\ **Post date:** [August 29, 2023, 7:52am UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/7 "2023-08-29T07:52:52Z")
Hi! While the question is definitely interesting I would put the answer in different terms. When looking at reconstructing the text from a vector it is a comparison to a (de-)compression method and not a transformation into a semantic representation of the meaning. And while vector embeddings have been around for a while I am not aware of any archiving app that uses vectors to compress text. This is a good hint that the task is definitely not trivial. Next, yes, training data and even the complete model with weights and parameters and whatnot are available in the open source space. Thus, yes, researchers have tried to recreate the embedded text for example using AI models trained especially for this task. They wrote nice and somewhat intriguing papers but these approaches come with a lot of strings attached which leads to results like “in theory it should be possible”. But from the perspective of practically developing actual apps that make use of such a technology the answer is a lot closer to a hard “no”. You can take a look at this previous discussion where the intricacies are spelled out in more detail: > [@Is it possible to convert vector to text?](https://community.openai.com/t/is-it-possible-to-convert-vector-to-text/312794/): > > Hello. I have developed Chatbot system with text-davincci-002 model and Pinecone vector database. However, there is need to convert vector from pinecone vector database to text. In other words, reverse of embedding. I have used text-embedding-ada-002 for embedding. Is it possible? ---
**Author:** ![zhan1130](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/zhan1130/32/32807_2.png) [@zhan1130](https://community.openai.com/u/zhan1130)\ **Post date:** [August 29, 2023, 3:12pm UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/8 "2023-08-29T15:12:15Z")
That is perhaps the original use of it in the context of the adversarial attack setting. That is not what I intend to do, I just used it to make a point that such conversion is possible. I am interested in observing certain properties of LLMs. ---
**Author:** ![zhan1130](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/zhan1130/32/32807_2.png) [@zhan1130](https://community.openai.com/u/zhan1130)\ **Post date:** [August 29, 2023, 3:17pm UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/9 "2023-08-29T15:17:39Z")
My guess is the compression part happens at the positional encoding stage? Given my rough understanding of how it works, I can see why it is such a lossy compression and not reversible. So from the second reply onwards, I relaxed the setting to discuss whether it is possible to convert the vectors of the individual tokens back. ---
**Author:** ![curt.kennedy](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/curt.kennedy/32/709249_2.png) [@curt.kennedy](https://community.openai.com/u/curt.kennedy)\ **Post date:** [August 30, 2023, 2:55am UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/10 "2023-08-30T02:55:49Z")
I think inverting embeddings to recover language can result in weird tokens showing up in the output, especially for LLM’s, but might do OK on your own custom models, with super-limited vocabularies. For example, here I am embedding a series of images to a 2D vector space. I can also pick a random 2D vector and get back an image. So the embedding is invertible! But look at the images below, especially where the shirt morphs into the shoe _in lower image_. This is the crud you can get with words, and get it really bad since words generally don’t nicely “morph” into each other. ![Screenshot 2023-08-29 at 8.48.36 PM](https://us1.discourse-cdn.com/openai1/original/3X/c/a/ca5850fa1bf8cad7fbe89d7483346b348f681521.jpeg) To put it another way … What is the word between “vector” and “GitHub” … what is it? It’s likely not even a word! Just as the images above morph into each other, and while they are morphing, they are undefined things trapped between states of existence. So inverting would look a lot like putting the LLM at temp = 2 and watching the garbage flow out. ---
**Author:** ![zhan1130](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/zhan1130/32/32807_2.png) [@zhan1130](https://community.openai.com/u/zhan1130)\ **Post date:** [August 30, 2023, 3:21am UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/11 "2023-08-30T03:21:23Z")
Hi, thanks for sharing. May I clarify how did you do image embedding in your setting? I presume it is not a transformer so is it something like a VAE’s latent space? I am okay with limited vocab size and non-readable outputs since i am just trying to make some observations. ---
**Author:** ![curt.kennedy](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/curt.kennedy/32/709249_2.png) [@curt.kennedy](https://community.openai.com/u/curt.kennedy)\ **Post date:** [August 30, 2023, 3:38am UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/12 "2023-08-30T03:38:37Z")
It’s the VAE latent space. Here is how it could work for you in the model: You invert your own limited vocabulary custom model. It might make sense given the limited vocabulary. Here is how it works for everyone currently (including probably you): You embed a bunch of stuff from normal text. You pick a random vector, and find the closest thing in your collection, and call this the inverse. Since the embedding engine doesn’t have the “/decoder” endpoint, you are essentially left with a hash function, and have to “guess” to get close to something. You could try spinning your own, and see what comes out of the latent space though! Just not through OpenAI’s models. ---
**Author:** ![zhan1130](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/zhan1130/32/32807_2.png) [@zhan1130](https://community.openai.com/u/zhan1130)\ **Post date:** [August 30, 2023, 4:35am UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/13 "2023-08-30T04:35:50Z")
Cool, I’ll try and see what happens. Thanks for sharing! ---
**Author:** ![Foxalabs](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/foxalabs/32/738355_2.png) [@Foxalabs](https://community.openai.com/u/Foxalabs)\ **Post date:** [August 30, 2023, 9:43am UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/14 "2023-08-30T09:43:31Z")
I was listening to Mr Wolfram the other day and he said that the vectors we humans find interesting are 1 in every 10^600 that are out there. The was the “Ahh” moment for me, at least if I understood his point correctly. Curt’s "what is the word halfway between “Vector and Github” sort of clicked at that point, sure, in a 10^600 solution space… I bet there is a pretty accurate block of a few hundred quadrillion “words” that are for all intents and purposes “exactly” half way 😂 Humans are bad at large numbers and exponentials, the fact we even contemplate reversing an embedding vector demonstrates this. Not a criticism of the OP, just an observation on the self importance humans place on their exploration of less than nothing. ---
**Author:** ![zhan1130](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/zhan1130/32/32807_2.png) [@zhan1130](https://community.openai.com/u/zhan1130)\ **Post date:** [August 30, 2023, 3:52pm UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/15 "2023-08-30T15:52:52Z")
Yeah, I also saw one of his interviews on a podcast and his Ruliad space idea. I can agree that space is infinitely larger than human natural language (or even including all conceivable abstract ideas), but I think it still depends on whether we have set a constraint on just using a tiny subspace of it. From an information-theoretic viewpoint, if the compression is not lossy, we can in principle convert it back to natural language, but I am not very familiar with the inner workings of transformer to say if that is the case before the positional encoding stage (positional encoding is unlikely to be lossless). ---
**Author:** ![\_j](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/_j/32/766292_2.png) [@\_j](https://community.openai.com/u/_j)\ **Post date:** [August 30, 2023, 7:05pm UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/16 "2023-08-30T19:05:43Z")
Receiving “vehicle damage pattern” Answer similarity to: police chase, intoxicated, tornado, hurricane, microbursts, orographic lift, squall, lightning Reverse to input: Map daily water temperatures of the Atlantic. ---
**Author:** ![dan.petrovic](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/dan.petrovic/32/428914_2.png) [@dan.petrovic](https://community.openai.com/u/dan.petrovic)\ **Post date:** [July 24, 2024, 8:05am UTC](https://community.openai.com/t/converting-embedding-vector-to-text/336783/17 "2024-07-24T08:05:58Z")
Yes. This is trivial. If you have vectors for individual tokens **and** know the model used to generate the embeddings you can then generate embeddings for each token in that model’s vocabulary and map the encoded token via vector embeddings as a match. embedded sentences are a different beast entirely.