These models face two major difficulties:
-
Image interpretation: Their ability to read and interpret diagrams, drawings, and graphs is still quite limited. The o3 model made a slight improvement in this area by enhancing interactions with images, but image-reading capabilities remain very restricted. The best approach is still to manually describe key points to the model instead of simply providing the image.
-
Context Window: When dealing with large amounts of content at once, the model becomes less effective (I’ve written a paper about this: Reasoning Degradation in LLMs with Long Context Windows: New Benchmarks).
However, this doesn’t mean ChatGPT isn’t useful for your work. My recommendations are:
-
Include a description beneath each image, highlighting all relevant details.
-
Use reasoning models (such as o4-mini-high or o3) instead of GPT-4o, as they handle large context windows much better.
-
To further enhance results, summarize less relevant sections if you want detailed reasoning about a specific part of the document, thus reducing the overall context window size.
By adopting these practices, you’ll notice significant improvements.