When using the API to send a PDF file as context to answer a question, how is the token count calculated? Does it identify the relevant text first and then use standard method to apply the token count to the context window? Or is there a more specific way it handles PDFs?
3 Likes
The exact methods used for retrievals by the assistants API is not documented, so there is no way to properly answer this question. You should make the assumption that the AI and it’s subsystems will either make a RAG retrieval OR a full load of the document (if the text fits into the context 128k limit) at it’s discretion. The assistants API is a beta system that is under development, so expect this to be updated in due course.
3 Likes