Help getting LLMs to list things EXHAUSTIVELY?

GPT builders
1

Anyone have tips on how to approach the “give me a list of X” type questions?

I found that semantic search is not enough for these, because of its “retrieve top-k” nature. If details are scattered throughout multiple pages in the document (or many documents), typical RAG + semantic search is not enough.

Example:

  • List all the requirements for X listed in this document (fails especially if the details of each requirement is long, and requirements are spread across multiple pages). At best, it will list 5 out of 20 requirements listed in the source bc it uses the top N chunks for the answer.
  • Summarize this document (gives a convincing answer, but doesn’t always capture all the key points before generating a summary)

Things I’ve tried:

  • LLM chaining
    • ex. “List all the requirements for X”
    • step 1: loop thru all chunks/pages → GPT adds to a list of requirements any time it sees a new requirement it hasn’t seen before
    • step 2: do a top-k search for each requirement in list from step 1
  • business logic (“give me a list of requirements for X” → filter chunks by metadata X → retrieval on the filtered chunks)
  • for my use case, there is a limited set of “list x” type of questions. I could scan the entire document and calculate it ahead of time, since it’s slow/expensive at search time.

Things I’m considering (but require a lot of research):

  • Create knowledge graph of the document (create entities, then create a graph of chunks linked together based on those entities and their relationships. Use GPT to create entities from the chunks in a data pre-processing step)
  • Agents. Incorporate ReAct type prompting/chaining. I heard these are hard to debug and unreliable for the most part. Would love more data points here.
1 Like
2

Interesting one. I am working towards something that tries to address a similar problem.

On this option, if you already have labels for your chunks in the form of metadata - which I assume for now have some sort of relationship with X (e.g. X is a child of a label) - why don’t you just extract the requirements from each filtered chunk and then concatenate the list of requirements into the long list? Once you have the long list, you might need a “clean-up” step, e.g. to remove any duplicates etc.

2 Likes
3

Here’s an idea that may ultimately fail because of the amount of real attention that an AI has, with an order of magnitude increase in embeddings cost:

LLM topical chunk marking

Provide an AI criteria that must be met as a signal of a cleaer change in topic, such as a task-focused transition to a different specification within a document

Provide an AI line numbered documentation as large input.

Highly cheap:
Output line number of transition to array

Highly effective until the AI won’t output more than 1000 tokens due to an AI company’s supervised training:
Repeat back document, inserting %%% at section breaks

That depends on how completely spread out contents are and how implausible it is for an AI to answer in any case:

“I have never used OpenAI services. Give me a list of prerequisites and methods to programmatically obtain an answer from a specialized Assistants endpoint AI.”