Can Not Add Files To Vector Store

API
,
1

Hello!

I am trying to utilize the Assistants API to retrieve data from files, and I am having issues with using the vector store.

I have tried uploading files both via the API and via https://platform.openai.com/assistants or /storage, and both methods leave me having the same problem.

I am trying to upload a txt file that is 153Kb. For the sake of the example I am going to use the web page as it’s more visual as to what’s going on. When actually selecting the file to load it, appears to be fine.

But upon hitting the attach button and the page updating, it shows that the upload failed.

image
image1034×126 2.04 KB

It then becomes impossible to delete the file from the vector store and I have to delete the entire thing to clear the file out.

Am I doing something funny here that is causing this issue? Is this a known problem?

Let me know, thank you.

6 Likes
2

On top of this, assigning files to a vector store is also buggy I guess. Files successfully reflected as uploaded are also not mapped to any Vector Store.

3

try changing name of the file. Instead of 500leads, try leads500 or leadfivehundred. I saw a post where it was mentioned that if your file name starts with capitol letter, it will throw an error.

4

Very weird situation.

I went ahead and tried that solution and unfortunately even with just the name “leads” I’m getting the same problem. What a mystery!

image
image1036×121 1.96 KB

5

It does appear to be buggy in more ways then one.

I’m also seeing that if I switch between vector stores where the file upload has failed in the Storage tab on platform.openai.com that the entire page freezes and I have to force it to close.

As mentioned trying to delete the file results in a spinning wheel forever. I just checked the developer console and it looks like it might be an issue with updating the file? Let me try this in a different browser…

image
image677×742 41.3 KB

1 Like
6

I am too facing the same issue; attaching files, uploading files using browser or API fails without any errors.
Are there any logs that can be checked; the API response just says error.

7

Strange!

I reached out to their support chat and provided details on 5/7, I have yet to hear anything back yet. I hope this is on their radar, as it makes Assistants unusable for me until it gets fixed.

2 Likes
8

I have ran into the same issue today, file is just a basic .txt with song info in it - same error in the vector store (file not supported)

Here is a bit of the content in the file, very basic text info

#	Artwork	Track Title	Artist	Genre	BPM	Rating	Time	Key	Date Added
1		Shenanigans	Beth Lydi		124.00	     	05:41	5A	2023-09-30
2		just lose it bootleg 	neumonic		135.00	     	03:50	11A	2023-09-30
3		 Do My Dance 	Kenny Oliver		125.00	     	04:36	11A	2022-07-18
4		Hercules (Original Mix)	GetCosy	Tech House	130.00	     	05:25	11A	2023-04-22
9

I have solved the issue in my case - it was an encoding issue. Take a look at your encode type @CoryF

The assistant seems to choke on UTF-16 files :frowning: (see edit)

image
image1164×480 57.4 KB

EDIT: Turns out the assistant can handle UTF-16 fine - my file is just improperly encoded. Instead of checking encode type, check if it is properly encoded.

➜  Desktop iconv -f UTF-16 filth_utf16.txt > filth_convert.txt
iconv: unexpected end of file; the last character is incomplete.
1 Like
10

I have the same problem, tried to change the encoding but it still fails to upload.

11

I still have the exact same issue. It is not working for one file but with an other one with similar content, its working fine.

Even weirder is that this is not only happening in the OpenAI platform interface and API but also in the Azure AI Studio with assitants and vector stores.

grafik
grafik1360×1173 59.5 KB

More detailed errors would be great there, to find out what is happening on our side.

1 Like
12

I actually got it to work now, turns out it blasted the token limit (5m tokens per file), even though file is only like 12mb. I batched 5000 rows at a time and uploaded 6 different files and it works perfectly now. This might be excessive but i have no clue how many tokens the original file had so i just tested with 5k and it worked.

Try to devide it in half and see if it works, if not make even smaller chunks.

13

On my side I got it to work now by encoding the files as UTF8 with Notepad++ (which also makes the file size smaller but leaves token count the same).

Previously it didn’t work even though my file size was just around 500kb but I don’t know to how many token that equals to.

14

Try copy and pasting the content of the file that didn’t work into a new text file. I found that the encoding didn’t matter.

15

it seems the thread is still posting, so I am adding what I have just found… When it fails, but if you try two to three times more, it completes uploading, which is so strange.

image
image1413×240 55.3 KB

1 Like
16

I was able to solved this problem - use ANSI encoding in txt file. I spent a several days to solve it…

1 Like
17

I’m having the same problem, I tried to upload normal images like .jpg files to vector store. It said unsupported, so I’m lost… what type of images can I use as tuned models? This is quite annoying.

18

I had the same problem that bothered me for a few days. I solved it combinig multiple answers.
First, note that i am using the C# api which is functional but not perfect. What i had done before is create the files then create new assistant with new vector store. This left me with multiple files failing to attach.

The solution was to use existing store AND WAIT for the files to be attached to the assistant, this is the code:

vectorStoreClient.AddFileToVectorStore(“vec_store_id”, file.Id, true);

The last parametar is set to true.
Additionaly i have converted the files to ASCII and removed some html tags that i had in my files. I also converted to file names to lower case as somebody suggested.

Hope it helps.

1 Like
19

Adding one by one works really well, I waited 200ms between additions too.

20

I ran into issues with adding files to a vector store as well.
After trying many different strategies, I gave up on it working the first time. After the file uploads and vector store creation were completed, I run this.

export const fixAnyUploadIssues = async (vectorStoreId: string) => {
  const filesPage = await openai.beta.vectorStores.files.list(vectorStoreId);
  const files = filesPage.data;
  const failedFiles = files.filter(file => file.status === "failed");
  console.log("Initial failed files", failedFiles);

  for (const failedFile of failedFiles) {
    let attempt = 0;
    let success = false;

    while (attempt < 5 && !success) {
      attempt++;
      console.log(`Attempt: ${attempt} for file ${failedFile.id}`);
      try {
        await openai.beta.vectorStores.files.create(vectorStoreId, {
          file_id: failedFile.id
        });

        const updatedFilesPage = await openai.beta.vectorStores.files.list(vectorStoreId);
        const updatedFiles = updatedFilesPage.data;
        const updatedFailedFiles = updatedFiles.filter(file => file.status === "failed");
        if (!updatedFailedFiles.some(file => file.id === failedFile.id)) {
          success = true;
        }
      } catch (error) {
        console.error(`Failed to upload file to vector store: ${failedFile.id}, attempt ${attempt}`, error);
      }
    }
  }
};