Forcing assistants to pull fresh data from tools every time

API
, ,
1

I’m using the Assistants API and some custom tools to fetch and do QA over data from my API. The output of my API can be quite lengthy, and not all of the properties are relevant in answering a question.

Assuming the API details books in a bookstore:

[
    { "name": "Some book", "year": "1986", ... dozens of other properties },
    { "name": "Some other book", "year": "1982", ... dozens of other properties },
    ... hundreds of other books
]

To reduce token usage, I have some function arguments which act as property filters. GPT-4o determines which properties are relevant to the query and we end up with tool output that contains only the relevant properties to answer the question.

This works great first time round. However, when I follow up with another message, the assistant seems to be referencing the existing tool output, which may not include all relevant properties to answer the follow up question. It’s also likely stale in my context.

Does anybody have any experience of prompting in the Assistants API to force the assistant to call a function every time its output is needed, disregarding previous outputs for the same function?

I’ve tried call 'get_data' every time you need it. Disregard previous tool outputs. and so on, but nothing. I can’t see an obvious way of doing this via the API, so thinking prompting is my only option.

The ideal:

  • How many books were authored in the 1980s?
  • Call: get_data(attributes: ['name', 'year'])
  • Some output based on data above
  • How many sci-fi novels are there?
  • Call: get_data(attributes: ['name', 'genre'])
  • Some output based on data above
2

I’m confused.

Do you actually want to force a data pull each time? In your controlled environment it functions if you are asking a question each turn.

However this assumption would not function in a real-world setting.

It seems that the issue here is that the model is using outdated context.

In this case you can simply instruct the function call to ALWAYS be called when a book-related question is asked.

3

This would be easy to do in chat completions using your own management: Strip past tool call and response as soon as they are obsolete.

Assistants’ attitude of hiding the actual contents of a thread from the developer makes such management difficult. List the thread messages; see that you cannot send DELETE message on what is not seen and has no ID.

1 Like