I am trying to upload multiple files into a vector store using the UploadAndPoll
API. While the process succeeds for some files, it occasionally fails for others with the following error:
404 No file found with id 'xxxxx' in vector store 'yyyy'
.
I’ve verified that the vector store exists, and the files seem to be created successfully. Could this issue be related to a limitation of the API, or is there something wrong with my implementation?
Simplified Code Snippet:
const vectorStore = await openai.beta.vectorStores.create({
name: vectorStoreName,
});
const filePaths = [/* array of file paths */];
for (const filePath of filePaths) {
const stream = createReadStream(filePath);
await openai.beta.vectorStores.files.uploadAndPoll(vectorStore.id, stream);
}
return vectorStore;
What I’ve Tried:
- File Existence Check: Verified that the file is present in the vector store.
- Retrying the API Request: While some files are uploaded successfully, others fail with the
404
error, even on retry. - New Vector Store: Tried uploading files into a newly created vector store, but the intermittent issue persists.
What I’d Like to Know:
- Is this error caused by any known limitation or timing issue with the
UploadAndPoll
API when handling multiple files? - Could the intermittent failures be related to concurrency, file size, or other specific factors?
- Is there a recommended approach to ensure consistent results when uploading multiple files?