Using Buffer or B64 string for createImageEdit API in nodejs

API
,
1

Hello. I’m building a backend for our web app that uses createImageEdit api. Ideally I would like to have users provide images to our backend in either Buffer or B64 format, then we pass it off to openai for processing. However I’m running into some issues while trying to use createImageEdit API.

When testing with local images, I can’t seem to be able to get Buffer or B64 format to work. The API keeps returning 400.

  const imagePath = path.join(__dirname, "../tests/corgi-dog.png")
  const imageData = fs.readFileSync(imagePath,)
  const maskPath = path.join(__dirname, "../tests/corgi-mask.png")
  const maskData = fs.readFileSync(maskPath)
  return openai.createImageEdit(imageData, prompt, maskData, 4)

fails

  const imagePath = path.join(__dirname, "../tests/corgi-dog.png")
  const imageData = fs.readFileSync(imagePath, "base64")
  const maskPath = path.join(__dirname, "../tests/corgi-mask.png")
  const maskData = fs.readFileSync(maskPath, "base64")
  return openai.createImageEdit(imageData, prompt, maskData, 4)

fails as well

I was only able to get it to work when passing in streams to the API like so:

  const imagePath = path.join(__dirname, "../tests/corgi-dog.png")
  const imageData = fs.createReadStream(imagePath)
  const maskPath = path.join(__dirname, "../tests/corgi-mask.png")
  const maskData = fs.createReadStream(maskPath)
  return openai.createImageEdit(imageData, prompt, maskData, 4)

Does anyone know if createImageEdit API can work with Buffer or B64 strings? Some forum posts seem to suggest it’s supported but I just can’t get it to work.

our stopgap solution currently is to write the Buffer to disc then read it to a stream.

function hackyStuff(buffer: Buffer, path: string) {
  fs.writeFileSync(path, buffer)
  return fs.createReadStream(path)
}

But that’s far from ideal. If anyone has gotten any format other than stream to work with the createImageEdit API and can share the solution that will be greatly appreciated.

2

Welcome to the forum!

As a sanity check, I see you are generating images, do you have any image generation credits on your account?

3

yes. the account has enough credits as I was able to generate images with streams

1 Like
4

Ok, image gen is not my strong suit, @PaulBellow for visibility or any Dall-E expert.

1 Like
5

It might have to do with the alpha layer and/or whether it’s 24 or 32 bit. Not entirely sure on B64 strings. I know for editing I caved and ended up using the OpenAI library. You might take a peek at the library code for createImageEdit to see how they’re doing it. Please come back and let us know.

Digging a bit deeper it seems there’s a lot of issues logged (with solutions mostly) on GitHub…

https://www.google.com/search?q=createImageEdit+site:github.com&sa=X&ved=2ahUKEwir1bnUj6OAAxUplIkEHR3JDRoQrQIoBHoECBcQBQ&biw=1472&bih=740&dpr=1.3

Taking a look at their new v4 SDK might be helpful too…

1 Like
6

Wow I am also trying to figure this out today, I would be surprised it doesn’t accept buffer like createVariation API. Any updates on this would be great !

7

my opinion: Auto transcoder.
just because i never trust others to do something, but i can trust my own machine.
If that makes sense.

8

Its interesting because they use the same object “Uploadable” for both createImageEdit and createVariation. I am getting createVariation to work fine with the same buffer object in Node JS however the edit endpoint keeps throwing errors, kind of difficult to know what is happening to be honest. Any ideas Paul?

1 Like
9

Weird. With two people having same prob, maybe it’s a back-end issue on their part?

Is there a submit bugs form/email?

10

Hey @mahmoodmustafashille you’ve been able to get the createVartion to work??? I tried using a buffer but it keeps returning an error 400. Please share how you made it work.

11

Thanks for chiming in everyone and @PaulBellow thanks for those links. It was super helpful.
Happy to report that I was able to get Buffer working by calling the endpoint directly with axios.

here is a code snippet for those that are interested:

import axios from "axios"
import FormData from "form-data"

async function createImageEdit(
  image: Buffer,
  mask: Buffer,
  prompt: string
) {
  const form = new FormData()
  form.append("image", image, {
    filename: "source.png",
    contentType: "image/png",
  })
  form.append("mask", mask, {
    filename: "mask.png",
    contentType: "image/png",
  })
  form.append("prompt", prompt)
  form.append("n", "1")

  return await axios.post<{ data: any }>(
    `https://api.openai.com/v1/images/edits`,
    form,
    {
      headers: {
        ...form.getHeaders(),
        Authorization: `Bearer ${your api key}`,
      },
    }
  )
}

so it does look like a bug in the openai nodejs library. I’m new to the forum. How do I submit a bug report for this?

1 Like
12

Hey cool you solved it, weird I tried the same thing and still managed to get the very non descriptive 400 error.

Here is the image I am using

test
test512×512 28 KB

This file is originally sent in a base64 string from a frontend

const mainBuffer = Buffer.from(mainBufferString, 'base64');
originalImageBuffer = await sharp(mainBuffer)
      .toFormat('png')
      .resize({ height: 512, width: 512 })
      .toBuffer();
originalImageBuffer.name = "resizedImage.png"

I am then passing it to the form and using axios to post exactly as you did. I am using the same file for main & mask for testing purposes and a random prompt.

Do you know what else it could be?

1 Like
13

Yes here is the code I used to make it work

app.post('/create-var', async (req, res) => {
  try {
    const bufferString = req.body.message;
    const numImages = req.body.numImages;
    const resolution = req.body.resolution;
    const fileExtension = req.body.extension;
    console.log(fileExtension)
    const buffer = Buffer.from(bufferString, 'base64');
    let outputFileBuffer;
    outputFileBuffer = await sharp(buffer)
    .toFormat('png')
    .resize({ height: 512, width: 512 })
    .toBuffer();
    
    outputFileBuffer.name = "resizedImage.png"
    const response = await openai.createImageVariation(
      outputFileBuffer,
      parseInt(numImages),
      resolution,
    );
    const urls = response.data.data.map((item) => item.url);
    console.log(urls);
    res.send(urls);
  } catch (error) {
    console.log("Error Occurred");
    console.log(error.message);
    res.send({error: 'error'});
  }
});

It is a little messy but it eventually worked. Let me know if you get the edit to work I still cannot figure that out using essentially the same method here.

1 Like
14

it looks like your image is a jpg file. The mask requires transparency so I’d try converting it to a png and try again.

15

Hmmm… If this works, then it means I’m not far off at all then. Thank you so much.