I am using open ai api to extract certain information from text file. I have few text file which are extracted from PDF files and token of that files are greater than gpt 4 model limit (128K)
Added logic to split document in chunk using different method , i am getting output from first chunk but when i provide second chunk it is not returning any value.
Problem is second chunk depends on first chunk so how can we handle this case.
Also tried to extract summary of first chunk and inserted in second chunk and calculated limit again using recursive function but that is also not working because summary extraction is giving summary context wise.
Any help on this would be appreciated.