Token Budget in File Search tool

API
1

I am using File Seach tool and have stored json files in OpenAI Vector DB. I saw that default chunks that could be retrieved are 20 and could be fewer. I saw a note giving reasons why retrieved chunks could be lesser than the number of chunks set by the me. Can someone understand what does token budget means in this scenario. Does it mean even though gpt-3.5turbo-0125 has context size of 16k tokens only could store 4000 token info from vector db + system prompt?

Link for Reference

Here is the information :
" Note that the file_search tool may output fewer than this number for a myriad of reasons:

  • The total number of chunks is fewer than max_num_results.
  • The total token size of all the retrieved chunks exceeds the token “budget” assigned to the file_search tool. The file_search tool currently has a token budget of:
    • 4,000 tokens for gpt-3.5-turbo
    • 16,000 tokens for gpt-4* models"
2

Your interpretation is correct.

2 Likes
3

If it is really true, I don’t understand the reasoning behind defining the chunk size (4096 max.) and chunk overlap strategy (max.2048) . For example if i have chunk size of 4096 as my default chunk size open ai always retrieves 1 or 0 chunk as it exceeds the limit set by them . :sweat_smile: