I would be sure to create the data frame outside (above) the lambda_handler to make it a global in memory, and not executed each time. Or what I do, is avoid the data frame all together, and the pickle is a dictionary, where the keys are the hashes into the dynamodb database, and the values are the vectors, converted to numpy arrays.
The loading hit should only occur once at cold-start if you do it this way, since as long as the lambda is warm, you won’t see the load up.
But I strip out the keys in the DDB and embedding vectors, and put them in separate vectors separately (as globals outside the handler). So when you find the maximum inner product, you simply find the index of this, and use that index to get your hash into the DDB to pull your text.
Also not sure how you are searching, but a simple maximum inner product search, using the dot-product, is all that is required if you are using unit vector embeddings (which is what ada-002 uses). Example below:
def mips_naive(q, vecs):
mip = -1e10
idx = -1
for i, v in enumerate(vecs):
c = np.dot(q,v) # Manhattan for possibly more speed np.sum(np.abs(q-v))
if c > mip:
mip = c
idx = i
return idx, mip