Assistance Required with Azure OpenAI Model to Return JSON Format Output

API
1

Dear [Support Team / OpenAI Support],

I am currently working with the Azure OpenAI API and am facing an issue when trying to get a response in JSON format from the gpt-4 model using the AzureOpenAI package. I have set up the code to send an image (encoded in base64) and a query to the model, but I am encountering difficulties when trying to receive the output in the required JSON format.

Here is the relevant code I am using:

import os
from openai import AzureOpenAI
import base64

api_key = "xxxxx"
endpoint = "https://azureope123.openai.azure.com"
image_path = 'vm_10.jpg'

def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")

base64_image = encode_image(image_path)

client = AzureOpenAI(
    api_key=api_key,
    azure_endpoint=endpoint,
    api_version="2024-02-15-preview" # Or the correct API version
)

response = client.chat.completions.create(
    model="gpt-4",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe the image in a json format?",
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"},
                },
            ],
        }
    ],
)

print(response.choices[0])

Problem:

  1. I am attempting to send a base64-encoded image to the gpt-4 model using the AzureOpenAI client, and I would like to receive the output in a structured JSON format.
  2. Despite following the documentation, I am unsure if I am sending the request correctly or if the model will return the output as JSON. I would appreciate your assistance in ensuring the response is formatted as JSON.

Request for Assistance:

  • Could you guide me on how to ensure the output from the model is returned as a JSON object?
  • Is there any specific configuration or parameter I need to set for the API to return the response in JSON format?
  • Am I sending the image in the correct format using data:image/jpeg;base64,{base64_image}?

I appreciate your time and assistance in resolving this issue. Please let me know if additional details are required.

Thank you,
Abdul Manaf PV