Is this a right JSON specs of image analyzing function?

API
1

I have This JSON for a tool that calls gpt4v for analyzing images, so the user can send more than one image, I am wondering if the tool spec looks fine:

{
      "type": "function",
      "function": {
        "name": "analyzeImage",
        "description": "Analyze an image using GPT-4V",
        "parameters": {
          "type": "object",
          "properties": {
            "prompt": {
              "type": "string", 
              "description": "A text description of the analyzing needed for the images."
            },
            "image_urls": {
              "type": "array",
              "items": {
                "type": "string",
                "description": "a URL of an image to be analyzed"
              }
            }
          },
          "required": ["prompt", "image_urls"]
        }
      }	
    }

Also, I am wondering if there is a source to learn about declaring specs like this!

2

Put the image_urls description in the same level as “type”: “array”.

Also, I am wondering if there is a source to learn about declaring specs like this!

You can check JSON schema doc. Just be aware that only a subset is supported by the OpenAI function calling.

1 Like
3

The API endpoint does a particular validation for arrays: the ordering is important.

Here for example is a tool (and message employing it or asking about it), where if the order of “items” or “description” were reversed, you’d get an API error.

tools = [{
 "type": "function",
 "function": {
  "name": "ai_memory",
  "description": (
   "function stores short phrases about user or chat state in long-term memory database.\n"
   "- memory phrases will be automatically injected in later chats for learning.\n"
   "- automatically store any facts or info that would be useful for AI later."
  ),
  "parameters": {
   "type": "object",
   "properties": {
    "memory_facts": {
     "type": "array",
     "items": {"type": "string"},
     "description": "Short one-sentence factoids to be remembered, in list array"},
   },
  },
 },
}]

messages = [
 {
  "role": "system",
   "content": ("You are ChatAPI.\nPretrained knowledge cutoff: 2021-09; today is 2024-04-11.\n\n"
   "Factoids from long-term memory: null"),
 },
 {"role": "user", "content": (
    #"Hi, my name is Jake Nielson. I'm a programmer at LightAI. I like skiing.\n"
    "In your ai_memory tool function, what kind of data can you store in an array (exact description, please)?")
 },
]

Also note the placement of the description, not within “items”. With this description, the AI is able to answer about “factoids”, not seen elsewhere in the function.

In the ai_memory tool function, you can store short phrases about user or chat state in the long-term memory database. These are typically concise, one-sentence factoids that encapsulate key information or preferences expressed by the user during a conversation. The purpose of storing these factoids is to remember and utilize this information in future interactions to provide a more personalized and context-aware experience.

or the AI using it:

tool_calls:
{“memory_facts”: [“Jake Nielson is a programmer at LightAI.”, “Jake Nielson likes skiing.”]}

You can simply modify this demonstration of “user memory” (yet to come to most ChatGPT users) to be names and descriptions about your URLs and their purpose. The multi-line main description can be exhaustive, especially telling the AI what to expect as a return value.

Then for your particular case: consider if the AI could comprehend a separate array that matches up the image URL with the unique image task to be performed on it.

The OpenAI documentation is poor. Here’s examples of other JSON data types accepted (in function spec format):

functions = [
{
    "name": "data_demonstration",
    "description": "This is the main function description",
    "parameters": {
        "type": "object",
        "properties": {
"string_1": {"type": "string", "description": ""},
"number_2": {"type": "number", "description": ""},
"boolean_3": {"type": "boolean", "description": ""},
"empty_4": {"type": "null","description": "This is a description of the empty_4 null property"},
"string_5_enum": {"type": "string", "enum": ["Happy", "Sad"]},
        },
        "required": ["string_1","number_2","boolean_3", "empty_4", "string_5_enum"]
    }
}
]

The name, if it is required, the data type, and the description is all the AI receives. You might want to use other JSON schema specs, like number ranges, but they will be ignored.

And finally you can nest objects - and with functions, the second layer of object loses descriptions.

1 Like