API Error code: 500 - Tool call with image upload conflict

API Bugs
,
1

Describe About the issue

  1. This bug happens when the model is sent with an image, if I only ask for the text it works correctly, and the tool usage is correct.
  2. No matter whether the token in this value [1024, 2048, 4096, 8192] all result in the same status 500.
  3. The code without the tools will work correctly, we have to try to check out the older version without tools. (Notice: The package has been upgraded but we have tried to downgrade but still have the same issue)
  4. I have a test around our development team and all results in the same issue.
  5. I have checked the status and it seems the API server is still running smoothly these days.
  6. We adapt the end token to force stop at “\n” will make it generate an incomplete sentence without getting the error code 500 in some results (Not always work)
  7. We further debug the code by removing the tools that pass to the model and the result is the same.
  8. We tried multiple different images to test and all results the same error

How to reproduce the error

As the project we are developing on is an open source, the error can follow the below step to reproduce

  1. Clone the GitHub repo “whats2000/CodeBRT”
  2. Go to the “Web-Search” branch
  3. Move to the “VSCodeExtension/code-brt” package
  4. Init the project with “npm i”
  5. Open VSCode debugger
  6. Click “Watch + Run Extension” to start up development mode
  7. Select the openai model gpt-4o or gpt-4o-mini
  8. Send a message with an image upload
  9. The error will occur both on the console and an error message model at the bottom right as the image shown

image

Related Code that raise the error

const streamResponse = await openai.chat.completions.create({
   model: this.currentModel,
   messages: conversationHistory,
   tools: this.tools,
   stream: true,
   ...this.generationConfig,
} as ChatCompletionCreateParamsStreaming);

The code is declared with related classes

  • OpenaiService
  • AbstractOpenaiLikeService
  • AbstractLanguageModelService

Extra Infomation

  • Platform: Windows (All of our team are window 10/11)
  • NodeJS Version: v20.11.0