I was hacking around with the new embedding models and hypothesized they were all inherited from the larger dimensional version.
This looks to be true.
import requests
import numpy as np
Msg0 = "Aoccdrnig to a rscheearch at Cmabrigde Uinervtisy, it deosn't mttaer in waht oredr the ltteers in a wrod are, the olny iprmoetnt tihng is taht the frist and lsat ltteer be at the rghit pclae. The rset can be a toatl mses and you can sitll raed it wouthit porbelm. Tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef, but the wrod as a wlohe."
Payload0 = {
"model": "text-embedding-3-large",
"input": Msg0,
"dimensions": 1024
}
Payload1 = {
"model": "text-embedding-3-large",
"input": Msg0,
"dimensions": 3072
}
HEADERS = {"Authorization": f"Bearer {YOUR_OPENAI_KEY}",
"Content-Type": "application/json"}
r = requests.post("https://api.openai.com/v1/embeddings",json=Payload0,headers=HEADERS)
q = r.json()
Embedding0 = q['data'][0]['embedding']
r = requests.post("https://api.openai.com/v1/embeddings",json=Payload1,headers=HEADERS)
q = r.json()
Embedding1 = q['data'][0]['embedding']
v0 = np.array(Embedding0)
v1 = np.array(Embedding1)
v1 = v1[0:1024] # truncate the large model at higher dims to match the low dim model
v1 = v1/np.linalg.norm(v1) # re-scale back out to unit hypersphere.
print('Magnitude of the Vector v0:', np.linalg.norm(v0))
print('Magnitude of the Vector v1:', np.linalg.norm(v1))
c = np.dot(v0,v1)
print(f"Dot Product: {c}")
# Example output showing similarity
# Magnitude of the Vector v0: 1.0000000393255921
# Magnitude of the Vector v1: 1.0
# Dot Product: 0.9999992894122343
So if you want to create an arbitrary dimensional version, within either large or small, you just take the embedding at the dimension higher, and then truncate/scale the vector to produce the desired vector.
So if you want a 512 dimensional embedding from 3-large, you just create either the 1024 or 3072 version, and just truncate and re-scale to a unit vector.
I confirmed you cannot mix large/small though, as evidenced here:
Magnitude of the Vector v0: 0.9999999908481855
Magnitude of the Vector v1: 0.9999999999999998
Dot Product: 0.010467056078529848
So this trick can only be applied within the model (large or small), not mixed between the two.
TL;DR You can save API costs by embedding once at the higher dimension, for a given model, and synthesize the other dimensions offline, including dimensions not offered by the API.
PS. Your MTEB scores will just gradually diminish as you truncate/scale, as expected, per dim dropped.
