Anyone else have this issue?
V1 API extracted text from PDF perfectly every time, but as soon as the update pushed, even without migrating to V2, the assistant is struggling very hard to extract the text required from the PDF.
We migrated over to V2 with Vector Store, and saw some improvement, but it was minimal, but still never correct all the way through. We have played with the temp and top, changed up prompt, to no avail.
Every single time, the first output does not extract the text required.
Oddly enough, run the assistant with another message that says something like “The text is there, you have done this several times before.”, and then sometimes it will come back with the extracted text.
I think we could probably reformat our PDFs to where the assistant can read this particular line better, but that is a difficult process in and of itself. Changing hundreds of PDF forms to suit this one need.
In our use case, this latest update has pretty much made the assistant useless. We can still use the assistant for chat, but no longer extracting data from our forms.
Did you try enabling code completion and telling it to use code completion to get the content. (You need to attach the file for completion as well of course ). That has worked for me.
I assume this is a temporary problem while they add the option to work with the text vs RAG only as it seems to be at the moment.
1 Like
Estou com problema parecido. Extrair as informações corretas já Ă© bem complicado, e estou constantemente ajustando as instruções. Refiz todas as minhas chamadas API, ajustando para a V2, e ativando o file_search. A princĂpio estava funcionando relativamente bem. De repente o Assistant começou a retornar respostas que nĂŁo tĂŞm absolutamente nada a ver com as instruções e nem com o arquivo que foi enviado.
1 Like
I’m not sure why code completion would need to be enabled, but we do have file search enabled.
But I continued to play with the instructions a bit, and came with a solution that improved 100%, although still not quite perfect.
Originally our instructions contained an example output like this:
- Summary of Indemnitor Application:
Defendant: Justice Laws
Bond Amount: $25,000
First Name: Jane
**Last Name:**Doe
Phone:
Address: 123 Fake Ave Evanston, WY
Current Employer: Walmart How Long: 2 Yrs
Spousal Name: N/A
**Cash: **
Savings:
Yearly Wage: $35,000
Rent/Mortgage:
Total Owed:
Bank Name: Wells Fargo
But then it hit me while creating another assistant to try this instead:
Summary of Indemnitor Application:
Defendant: [Enter Defendant Name]
Bond Amount: [Enter Bond Amount]
First Name: [Enter Indemnitor First Name]
Last Name:[Enter Indemnitor Last Name]
Phone: [Enter Indemnitor Phone Number]
Address: [Enter Indemnitor Street Address]
Current Employer: [Enter Indemnitor Current Employer and How Long]
Spousal Name: [Enter Indemnitor Spouse Name]
**Cash: ** [Enter Indemnitor Cash]
Savings: [Enter Indemnitor Savings]
Yearly Wage: [Enter Indemnitor Yearly Wage]
Rent/Mortgage: [Enter Indemnitor Rent/Mortgage Payment]
Total Owed: [Enter Indemnitor Enter Total Owed on Home]
Bank Name: [Enter Indemnitor Bank Name]
This improved the output %100. It still misses some things, but I think we will get that to improve over time.
Check my reply to jlvanhulst in this thread. It may help with your issue as well.
I did something similar, but it included very detailed instructions. In my case, I believe that the difficulty lies in the structure of the PDF documents, which when they are converted to text (using OCR), the data is not presented with the same structure, but completely dispersed, which makes it difficult to determine which is the correct data. But at this moment, the problem is different. LLM is returning random responses, which have no relation to the instructions or the document that was uploaded, and frequently returns: “I see that you have uploaded additional files. I will review them and get back to you with the results. Thank you for providing the additional information.”
1 Like
Yes, we also have detailed instructions. With the example output from my other comment included in the instructions.
It took me awhile to get the instructions right to get good outputs.
What has worked for me the best is actually going to ChatGPT and telling the Chat to provide ideal instructions that will work for what we are trying to accomplish with the assistant. ChatGPT then provides the instructions that we have been using for the assistant.
I got this idea from how the Chat GPT store. In the store whenever you create a GPT model, it takes what you type in for what you need, and then creates a prompt - to create a prompt that works for the model.
In fact, after each extraction the vector story, the file and thead are deleted.