Methodological information about embeddings

API
1

I’m curious if we know anything about how the embeddings being outputed from the /embeddings endpoint are being generated.

A lot of past work on embeddings is at a word-level. So, if you wanted to calculate the similarity of two texts (> 1 word each), you would use a mean embedding method and average together all of the embeddings for the words in the texts. It tended to be a really blunt instrument.

Does anyone know how these embeddings are being calculated, at least on a conceptual level? I was thinking maybe they were being grabbed from one of the self-attention layers, but even that seems to be on a word-level. Are they just drawn from a dense layer lower down-stream? Curious if we have any idea.

3 Likes
2

These embeddings work at the text level. So all the tokens provided are used to produce a single embedding, which is not a piecewise composition, like you’d do previously with word embeddings.

These models are specifically trained to produce a high quality similarity or relevance embedding of a particular size, rather than simply taking the weights after x-layers of a language model which predicts the next word.

5 Likes
3

@boris Is the approach some kind of contrastive loss with BERT-style CLS tokens?