Vision not working with OpenAI NuGet

API
, ,
1

I’m trying to use create an API using the official NuGet package from OpenAI (2.0.0-beta.5).
I want to upload a screenshot, and it should return the error code and description based on the screenshot.

[HttpPost("Analyze")]
    public async Task<ActionResult> Post(IFormFile file)
    {
        if (file == null || file.Length == 0)
        {
            return BadRequest("File is empty");
        }

        var filePath = Path.Combine(Directory.GetCurrentDirectory(), "uploads", file.FileName);
        using (var stream = new FileStream(filePath, FileMode.Create))
        {
            await file.CopyToAsync(stream);
        }

        var _openAIClient = new OpenAIClient("{apiKey}");
        var _fileClient = _openAIClient.GetFileClient();
        var _assistantClient = _openAIClient.GetAssistantClient();

        OpenAIFileInfo uploadFile = _fileClient.UploadFile($"uploads/{file.FileName}", FileUploadPurpose.Vision);

        Assistant assistant = await _assistantClient.GetAssistantAsync("{assistantID}"); 

        AssistantThread thread = _assistantClient.CreateThread(new ThreadCreationOptions()
        {
            InitialMessages = {
                new ThreadInitializationMessage(
                    [
                        "Hello, assistant! Please identify the error code and error message in this screenshot. Format your response as follows in json:{Error code: {code},Error message: {message}}",
                        MessageContent.FromImageFileId(uploadFile.Id)
                    ]
                )
            }
        });

        var responseText = string.Empty;

        ResultCollection<StreamingUpdate> streamingUpdates = _assistantClient.CreateRunStreaming(
            thread,
            assistant);


        foreach(var streamingUpdate in streamingUpdates)
        {
            if (streamingUpdate is MessageContentUpdate contentUpdate){
                responseText += contentUpdate.Text;
            }
        }

I’ve tested the screenshot in playground with that exact prompt, and it works perfectly.
But in the streamingUpdates, it successfully goes through: RunCreated → RunQueued → RunInProgress → RunFailed.
The error message in LastError is “Sorry, something went wrong”

1 Like
2

Happens for me too.
My assistant doesn’t consume images anymore. Doesn’t matter from API or in a playground.
Responds with: " Run failed. Sorry, something went wrong."

1 Like
3

It appears that the issue arises when attempting to read from openaifileinfo. My workaround was to upload the file to a third-party location (such as Azure Blob Storage in my case) and use that URL instead of uploading it to OpenAI’s file client.

4

the api in unstable, probably your code has nothing wrong