Welcome to the Forum!,
Will you be storing the 1Mb txt file repeatedly or just the once?
If the contents of the file are not going to change, at least not rapidly, then it’s a fairly trivial task with any of the pre built solutions out there, Weaviate, Milvus, Pinecone, Vespa ai, Chroma, Nomic Atlas, Faiss, Qdrant, Supabase to name only a very few.
Welcome! I highly doubt that for a long document the embedding would be accurate or not, embeddings usually work well with short text. But you can experiment on it a bit.
yes … single time file upload but its taking so much time to create embeddings !! is there way to improve
Ok, which vector database have you decided to use for your project?
pinecone database … acutally i had also tried by sending chunks in batches but its giving me rate limit error.