I might have to check out Redis.
Another approach that might work for you is to hash each text entry and store it in a database of at least Hash/Text/(Vector. optional). Then create a data structure in memory with only Hash/Vector. Search over all vectors in memory, get the closest N, and return the Hash values. Then retrieve the Hash values in your database. I can get less than 1 second latency with 400k items (embeddings) in a severless environment with this approach. My search was naive too, only looking at dot products since my embeddings are unit vectors.