Our company processes medical forms, and our users have to do a lot of manual transcribing. We’re trying to write an assistant that will try to pre-process a form before sending the results to medical staff for validation, but the assistant keeps giving back fake data from the form that was uploaded.
You can test this out in the playground if you take something like a 2792 PDF form, convert your filled-out form into images, attach the images to the assistant, and ask it to give you back the questions along with the filled-out answers. Make sure to set the model to something like GPT 4o that has vision capability.
It just gives back fake names, dates, etc. I tested out in the normal Chat GPT client and attached my test form data images to it, and it returned everything just fine.
I’m stumped as to why it’s doing this in the assistant API.