You can add to the “concern” pile: only one PDF being utilized at all - the last one in a role message of typed parts.
This on Chat Completions; I demonstrate switching the position in a user message content of two PDFs switches the one that can be correctly answered about: