Hi is it possible to let the API automatically truncate search files to the token limit instead of throwing an error, such as
InvalidRequestError: The document at index 54 is 203 tokens over the length limit of 2027.
If not, then what am I supposed to truncate my files to in advance?
I guess I need to estimate the maximum query length I will get and then truncate the docs to
2034 - max_query_len ?
metadata to keep track of how docs relate to eachother (ie paragraph 1, 2, of same page).