Context window in assistants

API
1

I’m using function calling in the assistants API. My assistant should build a query based on a model to answers the user question. My table contains weekly product sales so the table has a lot of records (even when filtering by date, for example) so I am constantly getting

"error": {
    "message": "'tool_outputs' too large: the combined tool outputs must be less than 512kb.",
    "type": "invalid_request_error",
    "param": "tool_outputs",
    "code": "invalid_value"
  }

Do you have a recommendation to avoid large context windows?
This is an example of a question

What product are growing in week 21 vs week 20 in 2024?
2

This may not fit your use case, but I generally break up the database into views so that the number of rows returned are more manageable and the data are partitioned more by function (almost like a “canned” query). I have not encountered this in my use cases so far, but I suppose if there’s still too many rows returned, then you may need to get your assistant to ask for more information to further filter down the rows. I’m guessing some state information may need to be stored on the backend to deal with follow up queries. This topic interests me, so I’ll be following how you and others approach this problem.

1 Like
3

When you submit your tool outputs are you measuring/keeping an eye on the size? Make sure you don’t return more than 512kb in your tool outputs and if so maybe make your functions more focussed, possibly functions to allow your assistant to fetch any data in batches?

4

My temporary solution was to tell the agent to paginate the search (heavy DB operation) and use with statements. @fluxtah yes, but some queries return more than 512kb, adjusting the function outputs to 512kb might not be the best due to the calculations (will try it though) and also will try to get the data in batches.

5

Cool I also hit that error and had to solve it in a similar way.by reducing the tool outputs size and it fixed the issue. Good luck!