Should a Custom GPT be able to count the number of items in a JSON list?

GPT builders
1

I created a Custom GPT with a single uploaded file, which is a JSON file containing a list of dictionaries. Each dict in the list has the exact same keys. The file itself is only 51KB. It acts as a kind of index for getting the right ids to call my API with. Anyway…

There are 260 items in the list. When I ask GPT how many items in the list, the first answer I got was 204. I asked it to check again and it got 158. I asked it to count the number of keys labeled ‘item_id’ and it again gave me 158.

Any idea why this would happen? What can I test/try?

EDIT: I can get it to recognize 40 items, but not 60. I ask it to calculate several different ways and it gets 40 when there are 40, but 50 or so when there are 60.

2

This sounds like a typical example of “LLMs can’t count”.
Would a simple solution like using a tuple as a key work for you?

I asked ChatGPT to write a quick example, but there are more solutions that achieve the same thing:

You can use tuples as keys in your dictionary, where one element of the tuple is the index and the other is your original key. For example:

my_dict = {(1, 'key1'): 'value1', (2, 'key2'): 'value2'}

You could then ask to return the max value of the tuple. But I guess using code interpreter would be a straightforward solution also.

1 Like
5

The way for this to work: include code interpreter. Have the AI instruction work solely on files within the code interpreter environment if you need a count. 40 items is enough that the AI might not even be able to replay that accurately when producing a token-sampling based response.

1 Like
6

Thanks, I’ll try tuples. I also thought about simply adding another key/value: index/1…260 or maybe the key ‘count’. I could also simply include the known total in my JSON file, which I’m sure the GPT will pick up.

If I ask ChatGPT to give me a numbered list of the items, it gets them all and then it knows the total. I tried to get it to pre-count the items (invisibly) in the custom instructions but that didn’t work.

Ironically, the count is not important in my GPT, I just asked it how many as a test that it had all my data.

Thanks for you ideas and help.

1 Like
7

I have code interpreter turned. I am trying to avoid including it since the rest of my GPT has no use for it. And I don’t want my GPT to start using it for some random question the user might ask, which would like just confuse the user.

As I mentioned in a reply above, ironically, I don’t actually need the count, it was more of a test.

I will likely try having the first call to my API simply return the JSON data rather than have it as an uploaded file. I’m not sure though if that will make a difference in whether it knows how many items there are.

Thanks for your help! And thanks for that link, I’ll read through it more tomorrow when I try these other solutions.