LLM inference counting boolean flags from an api response

I’ve been trying to make chatGPT be able to count incidences of one property using agents and function calling (“which it works”), although the response is not analyzed correctly.
Response is simple, just a list of objects with one property:

[{"BranchIsDown":false},{"BranchIsDown":false},{"BranchIsDown":true}]
The prompt is fairly simple: “How many branches are down?”.

I get a random count every time I run the request.

I am confused on how should I improve my prompt so that the LLM can count constantly with the right count. Is this a Prompt problem? or I must fine-tune the model to fix this?

I am ok enhancing the context for this to work, then only thing I want to avoid is to over engineer the prompt, so that the main purpose of inference is lost and I get back to coding fixed functions.

I also want to avoid creating function tools for every case, I asume the purpose of LLMs in general is to understand and infer from non pre-set rules.

LLM’s at this point aren’t really great at counting things. So you might want to use your own code to vet the json response.

1 Like

Right, something similar to other operations. I just hope this will be resolved eventually. It just feels its a basic simple operation to work through financial and BI data.

Considering that the LLM uses tokens and not words it might be a big leap for this to be “resolved”. Also, please note that feeding financial data etc into an openAI product is not generally safe as that data will be stored somewhere and might be used for training at some point. Rule of thumb, don’t upload any personal, proprietary or any other information that might be compromising to your business to any LLM model that you do not directly control.

1 Like

I realized, that doing operations on my side works. And passing around private data is also not very accurate and efficient.

I think the best take away is to let the LLM infer the prompt for function calling, and let our server do the maths and counting, then let the LLM construct the end response.

Now I need to figure out a way to cache the private data, and make it available for the agent’s next steps without passing around the info as observation. (Obviously saving some tokens too.)

Maybe using an id to map the response and use it on demand?

I already tried to resolve my problem in one shot, but this reduces flexibility and needs over engineering for multiple possible combinations.

i.e.

  1. Bring me the store sales,
  2. Bring me the count of new stores.
  3. Bring me the sales new stores.
  4. Give me the average sales of new stores.

You get the idea…