BUG: Stored Chat Completions not showing in Dashboard when sending type "image_url" messages

API Bugs
, ,
1

Hey.

I’m currently trying to eval my chat completions, but they are not showing up in the Backend Dashboard. After further testing, I could reproduce the Issue with 2 simple commands:

Without Images:

curl 'https://api.openai.com/v1/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: ••••••' \
-d '{
    "model": "gpt-4o",
    "store": true,
    "messages": [
        {
            "role": "developer",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "say hello"
                }
            ]
        }
    ]
}'

This example WILL show up in the dashboard instantly. But when sending the same request with an image, it doesn’t show up:

curl 'https://api.openai.com/v1/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: ••••••' \
-d '{
    "model": "gpt-4o",
    "store": true,
    "messages": [
        {
            "role": "developer",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
                    }
                }
            ]
        }
    ]
}'
5 Likes
2

@edwinarbus could use some help on this. I met with @jannik to review this and confirmed:

  • in platform only TEXT content types will log into that chat eval dashboard
  • there seems to be no documentation on that being the designed / enforced behavior…
  • my assumption is that the goal of that dashboard is true just for text chat completions and not to use as a audio, image or other modality eval experience

would love to get the documentation here improved if we can. I’m sure @jannik would have some good suggestions!

https://platform.openai.com/docs/guides/evals

3 Likes
3

Hey, thanks for confirming the Issue was a great talk with you last Friday. I would love for this to be fixed / implemented in the future and until then:

  • Document it in Create Chat Completion - Store to be clear that these requests wont be stored.
  • Extend the Guides Here and Here
  • I would expect a Bad Request when sending image inputs with store: true, but I get that this is probably a bad idea to implement now as it would screw existing requests. Some feedback that something didn’t work as expected would be nice, tho. It took me some time to see that it’s not my fault as I didn’t get any errors

For my application, it would be crucial to improve results over time using the fine-tuning and eval features, which currently isn’t possible when using image inputs.

Would love to hear the opinion on this from @edwinarbus

1 Like