Banana != chocolate - having robot issues - need help from fellow humans

API Bugs
,
1

I’m using gpt-4.1 to identify food items from photos. My prompt instructs the model to visually identify foods in an image and return ingredient names

The issue: When I send a photo of a white bowl containing pieces of chocolate and blueberries, the model consistently returns “blueberries” (correct) and “banana” (wrong — these are clearly chocolate pieces).

This isn’t random — it’s reproducible across multiple calls. The prompt emphasizes visual inference and says “Only identify food that is CLEARLY VISIBLE in the image.”

A few things I’m wondering:

  1. Is this a known issue with gpt-4.1 and dark-colored foods?

  2. Would a different model (gpt-4o, etc.) handle this better?

  3. Any prompt engineering tips to reduce food identification hallucinations?

  4. You get what you pay for?

Thank you for taking a look and any suggestions!

1 Like
2

Perhaps you are not able to upload the image - it might be useful to share it? But as a new member perhaps that is restricted.

Stick around and contribute and you will gain trust level sufficient to upload images.

1 Like
3

Welcome to the dev community, @BotBot.

Can you share the API call you’re making to the model?

Without looking at the request, I can only advise setting a low value for temperature and/or trying gpt-4o, or using a reasoning model from the gpt-5 series.