Not able to upload file from client

API
1

I’m struggling to upload a file directly from the client. In the documentation the create function uses fs.createReadStream(filePath) to create a stream of the file (which uses the file system) but I need to send the file selected by the user via the file select input field so the file never goes to the file system. So I send the file as FormData() to the endpoint and then try to create a read stream of it but I get this error that it does not support it. I have also tried to directly set the Buffer i get from the client to the file key in the create object but then I get an error saying that the value transmitted exceeds the capacity limit even though the file is not too big (it is 43KB and the limit is 512MB):

The buffer directly:

const file = await openai.files.create({
            file: fileData // Buffer from the client,
            purpose: "some purpose here",
        });

Creating a stream of the buffer:

      const stream = Readable.from(fileData); // fileData is the buffer from the client

        const file = await openai.files.create({
            file: stream, // Pass the stream for the file upload
            purpose: "some purpose here",
        });

Is there a workaround / fix to this? Or should I write the file to a temporary folder and then create the read stream from there with the fs.createReadStream() or what is the best practise?

2

Hi Nicolas,

Yes for me, the best practice is like below.

const addedFile: FileObject = await openai.files.create({
    file: fs.createReadStream(file.path),
    purpose: "assistants",
});

You should pass file.path as a parameter, not buffer or file itself.

3

Have you find a way to transfer data without using a temporary file?

1 Like
4

Hey @nicolas.tuomaala00 , have you got any solution to this issue ?

1 Like
5

Wondering this as well.

Use case: I want to store files from a given url into my openai storage directly.

1 Like
6

this seems to work for me:

import * as streamifier from 'streamifier';

...
      const stream = streamifier.createReadStream(buffer);
      const file = await openai.files.create({
        file: stream,
        purpose: 'assistants',
      });
...
7

I’m getting this error: {
“error”: {},
“label”: “API Call”,
“message”: “Received null for "file[_readableState][awaitDrainWriters]"; to pass null in FormData, you must use the string ‘null’”
}

How are you passing the buffer in?

8

Yes, I am seeing the same error, my previous code was incorrect.

To make it work, I had to create a temp file and then do a
const fileStream = fs.createReadStream(tempFilePath);

Do let me know if you find a better way.

9

I’d like to solve it as well, went into a complete rabbit hole with streaming data over openai API

I’m trying to do it manually since the lib designed to work with local files or files in a browser.

import fetch, { File, FormData } from "node-fetch" // the most standard, native forms handling

  async uploadFile(
    stream: NodeJS.ReadableStream,
    filename: string,
    purpose: "assistants" | "vision" | "batch" | "fine-tune" = "assistants"
  ): Promise<any> {

    const fake = new File([stream], "taxes-1.jpeg", { type: "image/jpeg" })

    form.append("file", fake)
    form.append("purpose", purpose)

    const headers = {
      Authorization: `Bearer ${ApiKey}`,
      "Content-Type": "multipart/form-data"
    }

    const response = await fetch("https://api.openai.com/v1/files", {
      method: "POST",
      headers,
      body: form
    })

This finally makes correct request, but the content of the file is not there.

NodeJS.ReadableStream is what S3 client in AWS SDK v3 returns.

I’m really trying to avoid reading stream completely into memory as a Blob or Uint8Array, or god forbid save it on disk and load as the OpenAI library suggesting.

10

Let’s step back. OpenAI implemented api endpoint to upload files with form data. Which is a front-end friendly solution. All these attempts to make it work on server side are workarounds for the form uploads.

Openai folks, do you have plans for a scalable storage? E.g. chunked uploads.

11

Using MDN NodeJS native “Request” type:

/**
 * @param {Request} request - The incoming request object.
 * @returns {string}
 */
async function getUserTranscription(request) {
    const formData = await request.formData();
    const mp3Blob = formData.get('file');

    const transcription = await openai.audio.transcriptions.create({
        file: mp3Blob,
        model: "whisper-1",
    });

    return transcription.text;
}
1 Like
12

this works for me

async streamToFile(inputStream: Readable, fileName: string, mimeType: string): Promise<File> {
    const chunks: Uint8Array[] = [];
    for await (const chunk of inputStream) {
      chunks.push(Buffer.from(chunk));
    }
    const fileBuffer = Buffer.concat(chunks);
    return new File([fileBuffer], fileName, { type: mimeType });
  }

example

const response = await axios.get(url, { responseType: 'stream' });

return this.openai.files.create({
      file: await this.streamToFile(response.data, 'name', 'application/pdf'),
      purpose: 'assistants',
});
2 Likes