I have almost 1 MB of txt file and i want to create embeddings. What is best approach to store in time optimized manner. i want to make it work for javascript .
Welcome to the Forum!,
Will you be storing the 1Mb txt file repeatedly or just the once?
If the contents of the file are not going to change, at least not rapidly, then it’s a fairly trivial task with any of the pre built solutions out there, Weaviate, Milvus, Pinecone, Vespa ai, Chroma, Nomic Atlas, Faiss, Qdrant, Supabase to name only a very few.
All of those should have premade boiler plate code for javascript integration.
Welcome! I highly doubt that for a long document the embedding would be accurate or not, embeddings usually work well with short text. But you can experiment on it a bit.
yes … single time file upload but its taking so much time to create embeddings !! is there way to improve
i want to know is there any specific solution to this in javascript !!
Ok, which vector database have you decided to use for your project?
pinecone database … acutally i had also tried by sending chunks in batches but its giving me rate limit error.
1 Like