Mask is completely ignored in the image edit api. Is this expected?
Unfortunately this appears to be a known limitation of the model. The mask serves as guidance only. Think of it as a way to point to certain areas in the image. But not a way of leaving parts of the image untouched. The model still regenerates the whole image, it just tries its best to redraw the unmasked area as close to original as it can.
https://platform.openai.com/docs/guides/image-generation?image-generation-model=gpt-image-1
Unlike with DALL·E 2, masking with GPT Image is entirely prompt-based. This means the model uses the mask as guidance, but may not follow its exact shape with complete precision.
Here is a test showing that it isn’t completely ignored.
No I’m talking about the api. Not ChatGPTcom. The mask is completely ignored.
thanks for the thread, but just to confirm, so chatgptcom is completely able to do inpaint. But for some reason there is no API to do that? Again, I dont care about the soft masking, soft masking is good enough for me. But right now the API does not mask at all.
API - edits - gpt-image-1
Input with transparency layer mask:
Add a green baby dragon in the same style as the kitty here.
DALL-E 2 would be pixel-perfect, unable to draw anywhere not marked by the mask.
Results
(add this to the AI models running super-slow right now…)
Employing a mask with an identical pattern to DALL-E 2 - pointless and disobeyed. Not even infilled properly (the transparent mask is drawn-under in black RGB, but that would be invisible to DALL-E 2)
FYI, here’s prompting alone - and no mask, which would make editing impossible on DALL-E 2:
Add a green baby dragon and a few other cute animals in the same style as the kitty to half of the boxes.
-
five-minute timeout
Doesn’t understand “add” means leave the kitty alone.
@nom1 ChatGPT uses the same model as the API, but here it is in the API as well. You might not have the mask set up right. Mask should be an all white PNG with empty/transparent pixels where you want to change the original image, white where you want to more-or-less preserve original.
mask
file
Optional
An additional image whose fully transparent areas (e.g. where alpha is zero) indicate where image should be edited. If there are multiple images provided, the mask will be applied on the first image. Must be a valid PNG file, less than 4MB, and have the same dimensions as image.
https://platform.openai.com/docs/api-reference/images/createEdit
“Add a cartoon cat here, keep everything else the same”
“Add a frog here, keep everything else the same”
I put the demo CLI in a repo here: image-edit-demo/README.md at main · erikmalk/image-edit-demo · GitHub
Thank you for sharing your code, work and experience!
This kind of mask editing works well when the objects are simple or when there are only a few, like drawings or cartoon characters. But with real-life objects such as humans or animals, the model often changes their appearance, alters the colours, adds extra objects, or struggles to place the replacement object in the right spot.
For example, in the test below, the model placed the toy in the wrong spot and even created a third hand for the person.
I hope OpenAI makes improvements to this in Image Generation v2 soon.
@polepole I totally agree, the way it works now is a major weakness. The only point I was making with the demo is that that it’s not “completely ignored” (or it would have picked a place a random to add the cat and frog). So using the mask feature with this model could be better than nothing in some cases, but we still have to wait for improvements to the model to support true inpainting.