Actions are hallucinating their values or using non-current versions of action api schemas

[michael_1]

I have noticed that many times when I am testing actions with different 3rd party APIs, that ChatGPT will hallucinate the values for some of the parameters despite them being clearly defined. For example, sometimes it will replace the defined path URL with a made up value or it won’t even send a property despite it being marked as required.

Some examples:

Paramaters: not sending correct values:

ChatGPT is showing page_value=5 in the request window of the UI’s debug tool, but the API spec correctly defines this as page_value=30. In a previous version of the api spec I uploaded many revisions ago, this was set to 5 but it had recently been modified and saved and updated to 30. It makes me believe that custom GPTs are caching different parts of instructions and action specs and unless you create a new GPT from scratch, it retains random sections of previous api specs as artifacts. Can anyone confirm if this is true and if there are any other workarounds besides starting from scratch?

See debug in chatgpt request:

See open api spec excerpt:

    "perPage": {
      "name": "per_page",
      "in": "query",
      "description": "Number of items per page. Defaults to 30.",
      "type": "integer",
      "default": 30

Secondly, it is sometimes not sending all properties in a request despite it being identified as required.

Here is the response and the api spec is below where it indicates “file” and “sheet” being required in the request. The GPT will retry correctly and will function.

components:
  schemas:
    DocumentRequest:
      type: object
      required:
        - template_id
        - data
        - file
        - sheet
      properties:
        file:
          type: string
          default: ""  # Empty string as default value
          description: File for generating the document.
        sheet:
          type: string
          default: ""  # Empty string as default value
          description: Sheet for generating the document.
# additional properties follow

I have also tried putting a default text value into the file and sheet properties such as “null” but that did not improve anything

There is a lot of potential with actions but it is frustrating when it will execute perfectly in Swagger, but ChatGPT will not know what exactly to execute and which values to send as default values in a pre-defined open api spec. Let me know if I am missing something here.

[mcarussell18]

I have same issue here i can’t find a way around it at the moment!

[elmstedt]

I noticed this myself tonight while working on a new action.

For some reason I can’t quite discern, whatever system behind the scenes that converts the schema into a working API call gets kinda muddled.

It seems to happen when there’s some sort of issue that prevents the API call from reaching the server, then any calls after that continue to fail to reach the server and the model will occasionally hallucinate.

The only outright hallucinations I saw in this regard are when there aren’t any parameters that need to be sent to the API endpoint. In those cases it would simply pretend to call the API and hallucinate a response.

I thought I was losing my mind, because no matter what I did I couldn’t get the action to hit my server despite it working fine in postman.

In an act of desperation, I copied the schema to a new GPT and when I tested it, it worked immediately.

So, what I ultimately found that works is to copy the schema, delete the action, and create a new action with the same schema. (The schema needs to actually be correct, of course, for this to work.)

Until I can track down any kind of additional information on this, my recommendation (and personal workflow) will be to create a “new” action any time I do significant revisions to an action.

I hope this helps.

[elmstedt]

One other thing to note,

I’ve noticed the GPTs don’t really respect default values.

If you have a parameter that must be sent to to an API endpoint and you want it to be a fixed value, you need to specify in the schema for that parameter it’s enumerated with only one option, like this,

      parameters:
        - in: query
          name: per_page
          required: true
          schema:
            type: integer
            enum:
              - 30

Of course that’s only if you want the value to be fixed.

[robert_at_thebolt.ai]

To me it seems like the GPT’s version control system goes wonky when actions are edited.

My workaround has been to delete all actions, add them back, and change the info.version value in the OAS schema.

When that doesn’t work, I start over and build a new GPT from the ground up.