Could anyone tell me what is wrong with my JSONL file?

API
1

I am trying to upload jsonl file for the purpose of fine-tuning. But after spending multiple days I am lost.
I constantly receiver error:

error: SyntaxError: Unexpected token 'e', "error: ENOENT" is not valid JSON at JSON.parse (<anonymous>…, text: 'error: ENOENT'}

This is the content of my training set:

{"prompt":"aaa ->","completion":" bbb END"}
{"prompt":"bbb ->","completion":" cccc END"}

The name of the file is trainingfile.jsonl. If I knew how, i would upload it here.

Here is code from my angular project that prepares the training file:

    let tf = '{"prompt":"aaa ->","completion":" bbb END"}\n{"prompt":"bbb ->","completion":" cccc END"}'
    var blob = new Blob([tf], { type: 'application/jsonl; charset=utf-8' });
    var file = new File([blob], "trainingfile.jsonl", { type: 'application/jsonl; charset=utf-8' });
      let currentFileUpload = new FileUpload(file, file.name, this.projectKey);
      currentFileUpload.url = this.projectKey // this is a key used by user
      const uploadTask = this.upload.pushFileToStorage(this.userId, currentFileUpload, "/trainingfiles/")
      uploadTask.on('state_changed',
        (snapshot) => { },
        () => {
          if (file.type.includes("jsonl")) {
            let path = uploadTask.snapshot.metadata.fullPath
            const storage = getStorage();
            const imageRef = refStorage(storage, path);
            getDownloadURL(imageRef)
              .then((url) => {
                this.firestore.addFileToFirebase(this.userId, this.projectKey, this.projectKey, "content", this.userId, uploadTask.snapshot.metadata.name, uploadTask.snapshot.metadata.fullPath, uploadTask.snapshot.metadata.contentType as string, false, "content").then(() => {
                  this.http.get("https://<fire project id>/uploadFile", { params: {userId: this.userId, filePath: url} })
                    .subscribe((response: any) =>
                      {
                        console.log("returned from function uploadFile "+response)
                      });
                })
              })
          } 
        }
      )

Or maybe it is the problem with my firebase function (I use firebase function to communicate with openai api):

exports.uploadFile = onRequest({ cors: true }, async (request: any, response: any) => { 
  const filePath = request.query.filePath
  if (!filePath) {
    response.send("Please send query parameter!");
  }
      const res = openai.createFile(
        fs.createReadStream(filePath),
        "fine-tune"
      ).then(() => {
        response.send(res)
      })
});

To summarize my algorithm goes as follows:

  1. Preparing string containing training set
  2. Saving training set as jsonl file
  3. Sending the file to firebase storage
  4. Adding record of the file to firebase
  5. Launching firebase function uploadFile with url of the jsonl training file in the fire storage
  6. in firebase function sending url through stream to openai with use of createFile method
2

As a new user (Trust Level 0) you are not allowed to upload files.

See:

1 Like
3

Please post more of your code. Particularly the lines involving JSON.parse.
I do not believe it is a problem with your JSONL file at all, that looks good to me. :face_with_monocle:

4

I added more code, thank you for help.

5

EN0ENT usually means that it can’t find the file or directory. Maybe the path is incorrect.

I’m not sure what’s wrong, but it might be this:

const res = openai.createFile(fs.createReadStream(filePath), ...)

The filePath is expected to be a local path but it seems like you are passing a URL. Maybe try downloading the file first, and then linking the path to that file in filePath

6

When I copy filePath to browser, I can download the requested file to my local drive. So I think this is not it, or I don’t understand the way it works.

7

It’s this. The function fs.createReadStream(filePath) does not accept URLs, only local file paths.

8

Thanks for pointing to root cause, but how to write the code point to local file in Google Storage in Firebase Function? Heh, I am lost.

9

You need to either actually download the file, and then point the read stream at that file, OR you need to use some networking library to create a read stream that comes from a remote URL.
Check the documentation for the libraries you’re using to figure out which is the right option for you.
Or, I guess, ask ChatGPT? :smiley:

10

I did it! With help of chatGPT ofcourse.
Here is working code if anyone need it.

exports.uploadFile = onRequest({ cors: true }, async (request: any, response: any) => { 
  const filePath = request.query.filePath
  if (!filePath) {
    response.send("Please send query parameter!");
  }

  if (!admin.apps.length) {
    admin.initializeApp();
  }
  const bucketName = "<firebase application>";
  const storage = new Storage();
  const bucket = storage.bucket(bucketName);
  const file = bucket.file(filePath);
  let downloadedFile;
  try {
    let tempLocalPath = "trainingset"
    downloadedFile = await file.download({ destination: tempLocalPath });
    let contents  = downloadedFile.toString('utf-8');
      const res = openai.createFile(
        fs.createReadStream(tempLocalPath),
        "fine-tune"
      ).then(() => {
        response.send(res)
      }).catch((error:any) => {
        response.send("error: "+error.code)
      });
  } catch (err) {
      response.status(500).send('Failed to read file');
      return;
  }
});
1 Like