We use AI on our website (BuonMenu) to automatically convert restaurant menus in PDF format to a digital version (we use the OpenAI API).
The problem is that items (for some documents) are extracted in the wrong order.
Even if you use the latest models and reasoning and even if we tried different prompts.
It seems that your API (or AI) keeps using the extracted text order (from pdf source code) instead of the visual order of the rendered PDF.
Even adding this to the prompt doesn’t fix the issue:
Preserve item order exactly as visually displayed; use the visual reading order of the page, not the order of the extracted text.
That works in ChatGPT but not in the OpenAI API.
We cannot upload the PDF here in this forum because it is not allowed. If you want to investigate this issue we can provide it together with the full prompt.
If it is imperative that you keep the rendered order you might have better luck if you utilize OCR. I frequently use Tesseract for OCR, but you have a lot of options available to you.
That’s not necessarily the optimal solution in the long-term for your situation, but if setup correctly it should be reasonably efficient, and it should resolve the error with minimal effort while you investigate the issues you’re having further.
You’d have no problem letting OpenAI use the OCR and do its work with that data.
content parts of a user message for “input_file” type of part, and on what API endpoint?
I have noted before that when there is searchable text in the PDF, the file attachment of a PDF to a user role message can have the promise of a “vision” component of the PDF, each page rendered as its own image, omitted.
If the task cannot perform without computer vision - input to models as images - then you should prompt the AI that if it receives only extracted text and not accompanying duplicated “screenshot” of a PDF per page, then it must emit an error message and not continue the task. That will let you see if you simply should be rendering PDFs yourself to the required resolution and detail setting for vision image input and the AI model.
None of API endpoint, API model, upload method and vision method is explained in the first post.
I set Luna to task after asking it to render the PDF to an image, run OCR on the high resolution image, and let the model run with the data. It created an HTML version of the menu in 1-shot. I then asked it to generate high-level instructions you can use to create the pipeline.
Render every PDF page to an image at adaptive high resolution, preserving page dimensions.
Extract native PDF text and coordinates when available, but treat them as supporting data.
Run OCR or vision analysis on the rendered page images, retaining bounding boxes, reading order, language, and confidence scores.
Segment each page into regions or columns, then group nearby prices, item names, and descriptions using geometry.
Ask the model to convert those ordered regions into strict structured JSON, including page, order, section, name, price, description, and source coordinates.
Reconcile native text, OCR, and vision results. Use the rendered image and coordinate consensus as the authority when they disagree.
Validate for missing items, duplicates, impossible price associations, out-of-order blocks, and low-confidence text. Send uncertain pages to human review rather than inventing corrections.
Test against a representative menu corpus containing scans, text PDFs, multiple columns, photographs, unusual fonts, multilingual text, rotated pages, and decorative layouts.
Give it a shot and let me know how it goes. You could also recreate the menu in a variety of other ways with this pipeline.
If I don’t hear from you again, I’m wishing you success with your business!
You should render each PDF page on your server as a high-resolution PNG, ideally around 300 DPI, preserve the original page order, and send those images to the Responses API as ordered input_image items with detail: "high". The prompt should tell the model to treat the images as authoritative, preserve column and section order, and avoid alphabetizing or regrouping items.
OCR is helpful as a supporting layer because it provides searchable text, character/word coordinates, and confidence scores. For clean digital PDFs, native text extraction with coordinates may be sufficient; for scanned or photographed menus, use OCR such as Tesseract. Retain TSV, hOCR, or another bounding-box format rather than plain OCR text, since plain text can have the same ordering problem.
Use a strict JSON schema containing fields such as page_number and visual_order, then sort, merge, and validate the results in your application code. The rendered images should remain the source of truth for layout, while OCR helps locate text, identify possible omissions, and flag uncertain prices or item names.
yes, you are missing something (why post a generic message like that?): you need to use the PDF that I provided to reproduce the issue, not a generic PDF
Using ChatGPT or using multi-turn agentic tool-using AI with a billed tool is not discussed here.
File attachment will have or should have a vision component, a per-page image, but also has a text extraction.
The failing is the PDF being scrambled. Observe the selection from the word “cola zero” and then only extending the selection to the next line “Coke”. What happens is previous item titles are selected, following item descriptions are selected, and then the pattern of what is selected changes at item price 6, which is inserted between “cola zero” and “Coke”. If you read the text extraction of the PDF, it would be extreme nonsense.
Then the opposite, starting the selection after the word “COCA” and moving up to select befor 0.5ml, which should have contained only 0.5ml, the price, and the start of the title:
The pricing never got selected. How about if I attempt to selecton only the price and the single number “6”? Look how far more, out of order is selected, and the selection also jumps to the page title.
Stuff is backwards. I could demonstrate with Acrobat Pro on another PC what this page looks like, but I think you get the point: that the items cannot be reconciled to a distant price as the minimum fault, unless there is image input to the AI model.
I would work with your graphic designer to ensure the correct item order in the internal PDF tree, for even re-annotation by Adobe Acrobat Pro will not fix this. Otherwise, you must use AI vision for images and reject any text extraction from the file attachment method of the Responses API.
I can reproduce what you describe in Chrome. But for example selection on Preview (MacOS) and Safari works properly.
Maybe OpenAI should update its strategy to use something more similar to what is used by MacOS, in order to deal properly with real-word documents that may not be perfect.
In any case I already suspected that, indeed I wrote this in my original post:
It seems that your API (or AI) keeps using the extracted text order (from pdf source code) instead of the visual order of the rendered PDF.
The question is: the PDF is malformed (note that it was uploaded to our service by a user, it’s not a PDF made by us), but why the AI cannot understand the visual order of elements from the rendered PDF? I mean, when you send a PDF file in base64 to OpenAI API, the AI model should receive both the extracted text and the rendered image, so why is it not able to describe the correct order of items?
Their software is in production. That highlights that they’ve both got an ongoing problem to resolve and a bug to report. I don’t work for OpenAI, but providing advice to help get prod back on track is something people are generally happy to chip in with. My comments fall into the latter.
