Function_call - multiple results in one call - unwanted memory effect

Prompting
1

I created a prompt with a function_call and get back the arguments from gpt4-0613 to call the function. The JSON format consists of a main key with an array as value. The array contains multiple json objects which count for multiple answers:

Blockquote
“arguments”: {
“products”: [
{“Full Name”: “Windows 10 for x64-based Systems”, “Architecture”: “x64”},
{“Full Name”: “Windows 11 for ARM64-based Systems”, “Architecture”: “ARM64”},
{“Full Name”: “Windows 11”, “Architecture”: “ARM64”},
]}

Basically, I provide a text with IT products in the prompt and want to get back a list of these products with their characteristics. The above example is very basic with only the architecture as an example.

In general, this works great but there is a weird effect which you can see in the example above. When a product name does not provide any further characteristic details (like the architecture), gpt-4 responds with the characteristic of the last item. Instead of skiping the architecture in the third item, it answers with ARM64 which it kind of caches. And no matter what I tell it in the prompt it is not changing this behavior.

When I change the order of the products in the prompt it consistantly uses the last information before the first item which misses this aspect like here:

Blockquote
“arguments”: {
“products”: [
{“Full Name”: “Windows 10 for x64-based Systems”, “Architecture”: “ARM64”},
{“Full Name”: “Windows 11 for ARM64-based Systems”, “Architecture”: “x64”},
{“Full Name”: “Windows 11”, “Architecture”: “x64”},
]}

So it is not really halluzinating, it just uses information from a former interation in a wrong way.
Instead it should just skip the architecture key in the JSON answer if no input for it is provided.

Can anyone help me with that or has had the same problem?

This is the corresponding prompt:

Blockquote
{‘model’: ‘gpt-4’, ‘messages’: [{‘role’: ‘system’, ‘content’: “You get text with information about products. Only consider affected product versions. Never add products with no affected versions. If no explicit version info is given, do not consider it. Generate an argument for the funcation call as JSON.dump(argument). Do not format in anyway and do without linebreaks, backslash n, tabs, indentations or whitespaces. Always omit properties without values. Be aware to never hallucinate!”}, {‘role’: ‘user’, ‘content’: “[‘Windows 10 Version 1809 for x64-based Systems’, ‘Windows 10 Version 1809 for ARM64-based Systems’, ‘Windows 10 Version 1809 for 32-bit Systems’]”}], ‘functions’: [{‘name’: ‘get_vulnerable_products’, ‘description’: ‘Get vulnerable products of a given text.’, ‘parameters’: {‘type’: ‘object’, ‘properties’: {‘Vulnerable Products’: {‘type’: ‘array’, ‘items’: {‘type’: ‘object’, ‘properties’: {‘Full Name’: {‘type’: ‘string’, ‘description’: ‘The full name of the affected vulnerable product exactly as given in the text.’}, ‘Architecture’: {‘type’: ‘string’, ‘description’: “Look for strings like 32-bit, 64-bit, x32, x64 etc. in ‘Full Name’.”}}}}}}, ‘required’: [‘Vulnerable Products’]}}], ‘function_call’: {‘name’: ‘get_vulnerable_products’}, ‘temperature’: 0}

Best,
Martin

2

Welcome to the OpenAI community @drm80

One thing that stands out a lot is that there’s a LOT of negative prompting in your system message - telling model what not to do instead of what to do.

1 Like
3

This might be beyond prompting, because the AI simply gets distracted from instructions by tokens it has seen. However, you can make your prompt and functions exacting.

(the function return message is not for when no value was just returned)

Cleaned to my understanding but not ran, with temporary pythonic triple-quotes so we can read it here:


{
  "model": "gpt-4",
  "messages": [
    {
      "role": "system",
      "content": 
"""
You are an AI software vulnerability checker.
Any and all software products named within user input is extracted and submitted for checking, but:
- only when a hardware platform cpu type or architecture can also be determined.
Omit all whitespace in function parameters.
"""
    },
    {
      "role": "user",
      "content": "user text" 
    }
  ],
  "functions": [
    {
      "name": "get_vulnerable_products",
      "description": "Match software products to vulnerable list, return vulnerable products",
      "parameters": {
        "type": "object",
        "properties": {
          "Vulnerable Products": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "Full Name": {
                  "type": "string",
                  "description": "Verbatim product name text extracted, with platform"
                },
                "Architecture": {
                  "type": "string",
                  "description": "CPU platform/bit depth"
                }
              }
            }
          }
        }
      },
      "required": [ "Vulnerable Products" ]
    }
  ],
  "temperature": 0
}

If you have a strict list of permissible “architecture”, you can use an enum specification.

Also, just noticed that, like your other conflated language, “vulnerable products” in your JSON is not actually what you’re submitting. It should be something like “extracted_software_products” for understanding.

Another “PS” - you don’t optimize tokens of functions, except their description. What the AI sees is completely different than what you send the API.

1 Like
4

Thank you _j!

I’ll try this out.

Can you please explain me, what you meant by “you don’t optimize tokens of functions, except their description. What the AI sees is completely different than what you send the API.” I could not get your intention here.

5

Thank you sps!
Will try to be more positive in my prompt. Good hint!

6

Sure.

You write a nice array, lots of indentation and formatting?

        "parameters": {
                    "type": "object",
                    "properties": {
                        "property1": {
                            "type": "string",
                            "description": "Example property"
                        },

The AI gets that rewritten into its own trained format by the API backend.

// Example property
property1?: string,

So make your code pretty, instead of obfuscated.

2 Likes
7

It is even not the AI, but the API that optimizes the JSON structures, I guess. That is why it costs you less tokens in the AI.

1 Like