BUG: Function calling not adhering to JSON schemas with `minItems`

API
,
1

In the example below, the function calling API is provided with a JSON schema with minItems equals 1. The response seems to ignore this requirement.

The schema provided to gpt-3.5-turbo-0613:

{'name': 'Game',
 'description': 'A game.',
 'parameters': {'$defs': {'Move': {'description': 'A move.',
    'properties': {'player': {'$ref': '#/$defs/Player'},
     'move': {'type': 'string'}},
    'required': ['player', 'move'],
    'type': 'object'},
   'Player': {'description': 'A player.',
    'properties': {'player_name': {'type': 'string'}},
    'required': ['player_name'],
    'type': 'object'}},
  'description': 'A game.',
  'properties': {'players': {'description': 'List of players',
    'items': {'$ref': '#/$defs/Player'},
    'type': 'array'},
   'moves': {'description': 'List of moves',
    'items': {'$ref': '#/$defs/Move'},
    'minItems': 1,
    'type': 'array'}},
  'required': ['players', 'moves'],
  'type': 'object'}}
1 Like
2

Hi,

I’m not sure if this is down to an issue with the support libraries or (as I suspect) the model is not returning at least one value, or possibly a combination of both.

A few things to try would be:

  1. Show the model an example of a correct reply and ask it to use that as a template.
  2. Tell the model to respect minItems = 1 is positive language i.e. use “do this” not “don’t do this”
  3. do not process the returned data right away, instead post process it with traditional software to correct any known errant behaviour with string manipulation.
3

The functions you construct are not passed to AI as a particular JSON type. It only knows to make JSON outputs (which are easily interpreted as python dictionaries) by the fine tunings that have informed the function-calling mechanism.

You seem to be passing keys and values that the API will just ignore when it constructs the language the AI sees.

You write a function, and what does the API look for and use at each nesting level?

Function

  • name
  • description
  • parameters

Parameters

  • type (no examples documented other than “object”)
  • required (parameters that must be returned)
  • parameters by name

Parameters by name

  • type (string, and possibly int, float, boolean)
  • description

Expect for each parameter the AI to only see:

– description
name (type)

Nothing else is going to be seen by the AI. If you try to make a second level of parameters, names better be good because there is no description passed either.

Your game descriptions are very terse. My intelligence is not so artificial, and I still wouldn’t know what value goes in “a move” unless we are playing battleship.

2 Likes
4

Hi @_j, thanks for replying.

You write a function, and what does the API look for and use at each nesting level?

Function

  • name
  • description
  • parameters

Parameters

  • type (no examples documented other than “object”)
  • required (parameters that must be returned)
  • parameters by name

Parameters by name

  • type (string, and possibly int, float, boolean)
  • description

Expect for each parameter the AI to only see:

– description
name (type)

Do you mean that arrays and their specifications are ignored? In the JSON schema in the above GitHub Gist, "moves" represents an array that specifies "items" and "minItems":

'moves': {'description': 'List of moves',
   'items': {'$ref': '#/$defs/Move'},
   'minItems': 1,
   'title': 'Moves',
   'type': 'array'}

Nothing else is going to be seen by the AI. If you try to make a second level of parameters, names better be good because there is no description passed either.

Can you please share a reference confirming that?

1 Like
5
1 Like