Need Expert Guidance: Building an AI-Based PDF Tagging System Comparable to Adobe's Implementation

Open Models
,
1

Hi everyone,

I’m looking for advice from people who have experience building AI-powered document understanding systems, particularly around PDF semantic tagging and document structure extraction.

Over the past four months, I’ve spent well over 100 API credits experimenting with different LLMs and prompting strategies to build a platform that can generate high-quality semantic tags for PDFs.

The goal is not simple OCR or text extraction. I want the AI to accurately identify and tag document elements such as:

  • Titles

  • Headings (multiple levels)

  • Paragraphs

  • Lists

  • Tables

  • Figures

  • Captions

  • Footnotes

  • Headers/Footers

  • Code blocks

  • Forms

  • Reading order

  • Other semantic document structures

The benchmark I’m trying to reach is Adobe’s PDF Accessibility/Tagging quality, but despite trying multiple approaches, none of the current models I’ve tested consistently achieve that level of accuracy.

Some of the things I’ve already experimented with include:

  • Different OpenAI models

  • Multi-stage prompting

  • Image-based document understanding

  • Text-based parsing

  • Hybrid extraction pipelines

  • Iterative refinement

  • Custom validation logic

Unfortunately, the results are still far from production quality, especially for complex PDFs containing multi-column layouts, nested tables, figures, and mixed formatting.

At this point, I feel I’m missing an architectural approach rather than simply choosing a better model.

I’m looking for people who have experience with:

  • AI document understanding

  • PDF parsing pipelines

  • Semantic tagging

  • Layout analysis

  • Vision-language models

  • Accessibility tagging

  • Production document intelligence systems

I’d really appreciate insights on questions like:

  • Is an LLM alone the wrong approach?

  • Should this be built as a multi-model pipeline?

  • Are there open-source models that perform better for document layout understanding?

  • How would you architect a production-quality PDF tagging system?

  • Has anyone successfully achieved results close to Adobe’s implementation?

I’m not looking for someone to solve the problem for me—I genuinely want to learn from people who have tackled similar challenges.

If you’ve worked on document AI, I’d love to hear about your approach, lessons learned, or even papers, repositories, or architectures worth exploring.

Thanks in advance!

2

I think your conclusion that this may be an architecture problem rather than a model-selection problem is probably the right direction.

I wouldn’t rely on a single LLM to infer the complete semantic structure of complex PDFs. I’d separate the pipeline into stages: deterministic PDF/layout extraction first, specialized layout or vision analysis where needed, LLM-based semantic classification after that, and finally a validation layer that checks hierarchy, reading order, tables, captions, and structural consistency.

That also makes failures easier to diagnose because you can evaluate each stage independently instead of asking one model to solve extraction, layout understanding, semantics, and validation in a single pass.

My own work is primarily in AI automation and multi-stage workflow architecture rather than PDF accessibility tagging specifically, so I can’t claim to have reproduced Adobe-level tagging. But from a systems perspective, I’d definitely explore a hybrid pipeline rather than continuing to optimize prompts around a single model.

I’d be interested to see what part of your current pipeline is producing the largest accuracy drop — layout detection, semantic classification, or reconstruction of the final document structure.