TLDR: I can’t figure out what the encoding format is for base64 embeddings returned by the create embedding API.
- We’re using the create embedding API (called via a Node script) to embed text and then store the result in a database.
- We’re specifying the
base64
encoding format to play more nicely with our database. - We’re failing to decode the embedding when we pull it out of our database (or directly when it is returned by the API)
Would love help from anyone who has restored base64 embeddings successfully!
Embedding generation
const embeddingInput = ... // this is a text field
const embeddingResponse = await openai.embeddings.create({
model: 'text-embedding-ada-002',
input: embeddingInput,
encoding_format: 'base64'
});
const [{ embedding }] = embeddingResponse.data;
return embedding // note that typing for this is broken when specifying 'base64' but that doesn't really matter here!
Sample embedding
Here is a sample embedding that was returned by the API that has failed our decoding attempts so far (you can try with this link):
H4aMvNPy6TvtIsS6UdrmvAvIG7xP/bQ8PujbvPsGl7yGBoG8maS3vHmaPz2Vl+67ZriRu53lGjzKfxG858C3PEwggzyG9iO864mJPOp5rDs+cEq84D7xPPC7/rx8l6s8hsIJvCeBU7vd6a080AVbvOdIprx7d/G7x5KCvN3prTu6ehW8CjDQu4ycp7w7c947HZpsO9N62Llenv87PizTu8p/ET0Rkly8aekXPAeHOLtScjI8AVnGPHwfmjtUg368P8SevDUhLz1bOd88HromvKdEk7woCcK89wmruzGcVDvHkgI8clzwPJKqX7wOxQe7SWeOu3AXirzhXqu8Ii+kvPXoAbyZ6C68B4c4vKYkWTw0ieO7qrmQu8f6Nj0Vn6U8f/xLPFUbyrx/dN06JRyzvDhSNTuq7ao66wEbPHZpuTuvt+s7NamdvBVrCzyia2Q7JaQhPcQNKLyia2S8uUr+OmlxBrzx2zg8IdvPO52xAD0+cEo8SwBJPNQigTz4GYg8MWi6u5jIdD1liHo73nGcvBWvArunAJw854ydvBiMNLvKK728xFGfu5N2RTtIA928bF6VPOHmmbyc1b28hn6SPFY7BDyniIq85qB9vMsHgDxFWsU64V6rvBDG9juS3vm8JFBNurzP2DxJq4U7Pvi4vCT8+DwXwM482nSwO5jIdLwU0z8710OqPMfGnLxliHo8rMpcPEt42jlHN3e6IdtPPMYe9Lzjb3c8zaC6vAHhtLz61n88jHxtPFKCDzesDtS7uUr+vL3ftTwrOkg8rdq5vG9LpDxRUni8Xq7cPLMIrLzRJZU7beaDvEmrBTsNHd87G3lDPHVZXDzNoDq8naGjvJ1tCbxStqk8ujaePILpWjujv7g7ZRDpPCVgqryTQqu8T8mavE4xz7w1dQO85ijsu3i+fDu25V07BSKYvAfLLzwKZOo83Q1rvPdNorz7Bhe9f4Q6PH8wZrwyeBc8VRtKPcDMRLxrPlu7WDzzPIDYDr3AVDM8OFK1vGmloDzZVPY8AI1gup+y77zEUR+/T0EsvSL7CTwLlAE8p4iKPCe17TwhpzU8AZ09vAN67ztIA108g4EmvHOMh7vmoH28jWiNvEUmKzlJ3x+8FWuLvFkoE7xgVoW8K/ZQPH2nCL15Iq48JWCqvMAA3zw7+0w80SUVvNQigTyPvVC7CNuMvHkirjxpLY+6XI0zPK2WQjwHQ8G89GATPZIyzjxv90+7W/XnO6mpszx4vvw8RSYrvPe11ru/NPk7deFKO1wVojxMIIM88ZdBPXY1n7yAUKA8ZdzOO9DB47oxJMM8w/3KvI1ojTu2KdU7LmvOu9MCxzynRJM7C4SkPMdOCzyvt2u8yjsaO7Y5sjmNJBa864kJvbDXJTzU3gm8XxIOuyGnNT0BJay6+m7LOh0S/jwYWJq8b8O1uKnNcDv7fii64W6IOzQB9Tqia+Q6qamzO82guryjvzi53i0lPCVgqrzetZO8Y4eLPKUU/LvuMqG8facIuh6qybtbOd88HRJ+vAhTnjzqJVi8fNuiOl6uXDynRBO9FBe3u5DNLbx9pwi9xqZiPBEaS7soCcK84JJFPIv0/jwy8Ki7k3bFu+hYA7xbBUU9K46cvGLf4jxzSBC8H8qDvIwUubzXu7u77Yr4vNHhnToNlfA8NWUmPLo2nrwAjeA7kBElO+ag/Tr+A4O8+tb/u0wggzz+azc8/nuUvI+tczxIi8s85qB9u9HhnTyTurw8H4aMvP0XYzsyNKC8mVBjPLTUkTvjb/c6p4gKvSVgqruztNc7VddSPPiRmbwntW08aR0yvQWqhjzx27g897XWvCHbTzyS7ta8QaHQvPAP07rN1NS6nW0JPFIuO7w0ieO7b7PYvJMOkbzxL428t0mPuxHW0zwCeQC9vaubulxZGbznOMm8fA+9PBgUIz2acJ28HiLbvGty9Tux54I8F8DOuytu4jpIm6i8HnavPGFXdLwE3iC68Lt+vC7j37ufwky87WY7PIhLZzsoTTk8yU96PDgemzutYqg8gBwGPSQMVjzq4eA8WbABvedIpjuzPMa8tFwAPMQNKL12AQU9dRXlO/FzBDqWtyg7maQ3vGKrSD1/QEM8N7rpPLSQmjwdmuy7ur6MvBewcTyMfO28szxGPUhH1DzK96K7WDzzvP7zJTyQiba8EeawvNpAlrw4YpI8zSipO61iKDmnAJw6xJUWu14mbrumrMc83aW2uY952btppSA8trFDu7pqODzq8T07x06Lu9RWG7sUf+s8C/y1PO0ixDyQzS08b2/hPO52GDwEAl67SFcxPDHgy7tvSyQ8Dj2ZPN75Cj0RTuW7QRliu6N7Qbw++Lg82uzBPLfBILxFrhk98FPKu/oqVDzeLSW803pYvPAP0zyFos+7OFK1uwsMEzxsgtI8mRxJPBWvAj1CbTa8lnOxPNlUdryfftU8G1WGu8wI7zw8Gwe7NSEvuRuJoLycGbW8beaDvNogXDyM0MG8VAvtujxPobtLREC89y1ovJBVHLz4GQg8o/NSvEJtNrzDhbk8AElpuR9ClTymnGq7idNVO5yBabq6JkG7HqpJu753gbxL8Gs8GBSjuT4c9rs4hk88vAPzuzs/xDtYPHM79wkrO+CSRbxcWRk8YFYFOuTTqLx8D708mWDAO+tFkrzAVDM8TqngvA6lzTrQweM6mRzJPAerdbw0zdq8EjoFPQFZRjzElZa7jWgNvElnDrxFaqI85NMovMFkkLsOLby8ry99vAEVTzy8A3M9grVAPVydkDvK58U87roPvEc3dzzeLSW9ve8SvMkbYDzDudO7eEbru1WjuDtvfz48bivqO9cPED3hoqI8Jy3/Oye1bbweuiY8GCSAvBHWU7yGwok8680AvFtt+TxF4jM8T/00O/rmXD0E3iA7thn4Oxpp5rwSOgU8C5SBPDKssbvt3sw8l8eFPBUnlDwHy688CmTqvLSQGrvA3KE8/RdjvNBZrzz0hFC7zaA6vZ+y7ztolcO8eM5ZvDzXj7zR4R08OMrGu/3jyDwEzkO76q3GvJkcSbzmoP08ZvwIPThiErxMzC47iZ87O2Izt7xC5ce8gmHsvOHmmTv+r668QaHQvL53Abz0YBO92nQwPNDB47tCBYI8EMb2uqY0NrqshuW8oAZEPF+KnzzKK727E/d8Oy8DGrs0AXU7KAnCPESOX7y3SY+8YiNaPKPPlTwOtaq8iSeqPDsLKru9ZyS8S4i3vBsRDzzgxt+7900ivJpwnTunzIE8Ij8BPMcKlDx1FeU8pwCcPMCIzbxMzC68iEvnOjgOvrzknw48fJervKoxojyc1b077v6GPJOGIjwK7Fg6s8Q0u4mPXrv+AwO7li+6PJLuVjy3fSk73IX8OlwVIjzuqjK859AUvBVrCzy6vgy8LBaLvJWX7jxoQW88Lp/ouD/EHjzE2Q08ZjCjvJwZNTyc+fo7ry/9O3atsDxski+87v4GPF++ubyvL3288aeevLSQmrtC9SS8Aa2avKdEE7zLB4C77rqPvFUbSjzUzqw7LRf6vLmeUjsR5rC8J4FTPGLvvzwfygM8TNyLOiSURLwDem87aA1VvLcFGLzgPnG7BzNkvNvIBD1p6Zc8OJasPDsLqrtmMKM7AI1gPCQMVru54kk8kmboO1mwAbzUmpK8szzGPHz/3zuwg1E6nW0JPLSQGjwbAbI8HiJbPMeSgrxIEzo7lvufuxsRD71vf746C7i+u7Y5sjoYFKO8rA5UvWzGSbsB8ZE8S0TAu/aleTw762+8ouN1PB/KA7ugFiE9NXUDvMp/kTxc0So5lutCvF9GqLx/hLq8Y4eLPLd9qTytYqi7bitqOuTTKLyx5wK8CmTqO/c9RbzTRj68PixTvCVwBzztEue8cuTevJAhgjwrwjY8tilVPEJ9kzxYxOG7L4sIPBicEb0khOe8SasFO16e/7utpp88dSVCPHY1HzyaPIM8kM2tu2KryLxbbXk8n35VvBvNl7xF8pA6UrYpPc3ksbtSLju99GATPEFN/DuAUKC84zvdvB66Jj04YpI8Y8sCuaq5kLy975I8LuNfvMc+Lj1oUUy8umo4uvuOBbwHu9I77jIhOxFO5bxmdJq8iEtnPNTeiTwRKii9s/hOPKa8JDzHChS8G82XPOBOTrxoyd08XOGHvOPDyzyMjMq7C1CKOoDYDryJr5i5AdHXPIlbRDzD/Uq8eL78vOTjBTzauCe748NLvF8SDrw7c966jJwnvHYBBb1S6kO6QaHQOeEqkTweIts6mVDjuvcJq7uFknK8BzNkunnukzzMCG88cNMSO5+y77zHgiU82VR2uxQHWjxP/TQ8SJsoPPuOBbyS3nk8eTKLOxR/azqA2A48uZ7SO1IuO712abm8XOEHPKUU/LxR2mY8CmRqvNQSJL0lLBC8EeawurRcgLqnAJy8Ra6ZOiQMVrxyKFY9zSipOwVmjzqzxLS8jATcu4lroboQxna7zjgGO+o1tbxr+mM8P0wNPEXiszt2vQ07mZTavKlVX71YxGE7s8S0Ow49Gb2zxDS9O4O7uu7uqTtTxoY4NXUDPFSD/jrnsNq7wJgqvOGiojzXh6E8lj+XPD60QbzoFIy81JqSvBSPyLvdpba7dmm5u8PJMLxcjbM84V4rvMwI77wuN7S8VRtKvMdOC73XhyG8bILSvHY1Hz2wGx09NInjPE7tVzrXu7s550imu9DBYzx+qHe8FFuuu/rW/zyshmW7sF8UvFSDfjy8z1i9kw4RO+cErzzz/GE8ReIzPMo7Gj0KdMe80wLHux66pjz41RA8T8mavFEe3rtM3Is8LifXvMRRn7xenn+8kIm2u2LfYrzu/oa82iBcPOFeq7xPUYm5iY/evHkyizwYSL27680AvAN67zwHQ8E85ijsumxelby9I607/ifAu9og3DsFqgY8yufFu9FpDLxMmJQ8ur6Mu4CUl7v7fqi84eYZO9p0MDweZlI7E/f8ua2mn7yzCCy8ed62POgUjDorjhy9eTILvNCNSbuK84+8GmnmO/Fjp7w7t9W8Qim/vHUVZTpZXC08uVrbvKrtqrxY1L47SANdvO2KeLzqeSw8YqtIPvpe7ru54sm7+46FPGWYV7yXgw48G1WGOjhiEjxCOZy8gwmVvKmps7zDdVw8reoWvXZ5ljvNKCm8SAPdO8Qdhb0KZOq80a2DvJARpTuz+M664eaZO5DNrTtIA927G1UGvNcPkLseuia8DoGQPO7+Bj3RJZU8secCveSfDjxiMzc8tqHmuy+/IjzJG+C6aMldPC43NDw4Qtg8b48bPI0kFrsUFze8Uj6YO9r8Hr1fAjG7CNsMPImvmLwsFgu9XibuvBjQKzwvAxq98Lv+O6q5ELu6vow86iVYPH2nCL2d5Rq8CB+EO8a2v7zHkgK8c3wqPARWsjzKo868cjizutlUdrwBaaM8xA2oPFtt+bwBJSw9neUavWlhqTzKw4g7tinVOrP4zjvnBK+8tm3MvONv97vTetg8OFI1u2Lf4jv3CSu8clzwO9AF27wfygO8MVjdvD4s07wH/8k80/JpPMBUM7z2pfm8FWsLvPTYJLytph+9eSIuO6BKuzqjAzC7c8AhPAhTHjz9F2M7lvsfPIOBpryg4oY9L4uIvFb3jLxJIxe94SoRvMeSAj0Hhzg9MdBuvN0N67sua047LFoCvTTN2jtWs5W8S0RAPNHhnTt/MGa82uzBvFt91rs4UjW8c4wHPUxUnbyA2I48SM/CO8f6Njx/DCm7BM7DPEFdWbzNbKA7lreovGUQaTy5ntI8yU/6PGv647yTypm8G0UpPCiRMD0FqgY9PJOYO23mAz1ybM27EdZTPGnZOjxR2uY7mSwmPDGc1LwfygM9+waXurcFmLypqbO7gnHJvPGnHr12rbA72dzkug6BED2tph87cuRevBDGdr3XD5A6LRd6vE7tV73AiE286jU1PLp6FToDem+8Oz9EvGySL77nBC88f8gxvBFeQjzbyAQ9raafPDV1Azrwu367wJiqu8lPeroSshY55J+OvLP4TjrgTk47w7nTPJzFYDwBrRq9Ne0UPe66j7s8k5g8ErKWPGs+27tmdJo7fNsivfct6Dxb9We7H4YMvSUsEDztmtW8ozdKvKmps7zQOfU8p8yBPF4m7jytLg49sINRO3NIkLvnfMA7s7TXO4jD+Dwq5vM857DaPBUnlLward081CKBvHYBhTxHN/c8vTOKPNf/sjwrfr+77Yr4u+ppz7z+A4M850gmvPWkijwrslk8Hv6dvArce7xvf767BM7DvA61KrxlZD08szzGPHIo1jr4GYg642/3O3Z5Fjugno86fqj3u8ZyyDyT/rO8c0gQvTv7TLxeJm68qd3NPLyLYbzXd0S9c4wHPUsASbw0ieM7VAttutAVOD15Iq47ZRDpums+2zwvv6I85wQvuzXdNzz+Nx09wmV/vDhikjvauCe93rUTPJ+ybzwbvTo7zvSOO5ZjVLpLeNo7swgsvKOLnjw8Gwc7OeoAPEzcC7mDTQw9gi1SPNNGvju9M4q8ZWQ9PEkjFz1sGp68ErKWvCiRsDyPeVk8JRyzPOQXoLt8h848DqXNO9dTB70Y4Ii7t8Egu4814jwlYKq8MryOO8fGnDwRGku8RI5fPD9Mjb1lVGA8aekXPAN67zzDddw73VFiuXXhSruCYew84/flO1tteTzNGMy7NBFSvJwZtbsO+SG83j0CPL2rm7wK3Hu7tNQRu6r9hzztivg85J8OvQQCXrx4Rmu8rR4xvNpAFr0a8dS5KF0WvAgPp7o3Mns89MhHPESOXzzN5DG8CJcVPJmkNzx+qPe7deHKPH/8S70Rbh86ASUsPXmaP722oWa3VSunO8SVFryjzxW9yjuau6q5EL0kUM28qu0qOv43nTucgem8WDxzu8SVFruqMaK7T1EJO07t1zweZtI7pngtvBickTxjy4K8k4YivCv20LuWc7E7+34ovYVeWDw++Lg8Rzd3Ox52Lzw4Dj68QV3ZPCct/zsrjhy80eEdPCK3Er30QNm77WY7vH2niLsIl5W8fJervL80eTqmeC06LFoCvIM9L737wh+9GJyRO09BrDy6vow8x8acOxuJILzgTs63QgUCvYbmRrv+8yU8yrMrPOeMHbySZmi8cvQ7uzhiErwUW667FEvRO8DMRLx/MOa8GzVMu0+Fo71M3Is8jBS5u/BDbbkhU2G7yqPOuT74ODwic5u8cNOSOMOFObs56gC91Io1PfBTSrydbQm7C7i+vMDMRDwbecM8UvogvGYwozxFWsW7i/R+OmIzN7w7Cyq8EdbTuK8vfTsV4xw9ErKWvAUimDwlHLO7LuPfO0jPQjztVt68Puhbu/4nQD2MfG28XJ0QvDcy+7vA3KE87v4GPS7j37yA2A68EMb2vJY/FzvW79W8tBiJvPr2OTugno+75/TRPD9MjTxIR1S848PLPCwWizwSOoW8YrslvONvd7zzdPO8k4YiPF4m7ru50uw8hZLyu9yF/Dxi/5w8+waXuyXomDsLDJM8IdtPPN2lNr2A2I67nPn6O1+KH73a/J68pvA+vGiVQ7w8T6G7f8ixPEVaxTyWt6g8wmX/uzSJY7utpp88UVL4PIORA7yvt+u8C5SBPNdDqjzqNbU894G8vLeNhjwngdO8EU7luk/JmrweuqY7NBHSuh/KA72pzfA7MqwxO71nJD0ntW270wLHPDv7zDzmKOw8MZzUPPoqVLxjQxS8o4sevVQLbTv+vwu9FWsLvfct6DsSspa65Eu6O16efzyGOhs8PnDKPI+t87o8kxg8GBSjOwiXFTj0yMe8BWYPPSjlhDpfNks8FH9ru/ctaLsLuD48g5EDPffFMzvO9I46T/00PMAQvLvpWXK87Yr4O0LlRzrX/7K8UnKyvBiMNLzbyIS8IvuJPHBbAbxWO4Q9LuNfO2NDFLyXxwU8sE83vAsMkzxfRig8wCAZu8am4jh9pwi9wIjNPCFT4TsYjLS6suhxvNdDKrzKKz28lZduPH6odzz+N528O+vvu4AchjzqaU88meiuPC9HEbw0EdK8maQ3vJo8A7wEiky7vVdHvC6f6LwSspY8oOIGvDJ4Fzwrfj+8/nsUPFzhh7yXxwW8ia+YvElnjjwOgZA8oJ6PvKlVXz0hy/K8+NWQvMqzq7xpLQ+5jHztvGWI+rvRrQO9