Optimizing GPT-4 Response Paging for Contract Data Retrieval

API
1

Hi, I’m currently working on a project where we extract key information from contract documents and store it in a database. Based on user queries, we retrieve the relevant data and send both the question and the data to GPT-4 to generate a final response.

Since the retrieved data can sometimes exceed GPT-4’s token limit, we’ve implemented a paging system. This allows us to pass a limited amount of data to GPT-4 in a loop while streaming the results.

For example, if we send a maximum of 10 customer contracts at a time, the response might look like this:

Here are the basic contract terms for all Customers:

  1. Client 1
  2. Client 2
    …
    …
  3. Client 10

Here are the basic contract terms for all Customers:

  1. Client 1
  2. Client 2
  3. Client 3
    …
    …

Does anyone have suggestions on how to improve this so that:

  • The introductory sentence (“Here are the basic contract terms for all customers”) isn’t repeated.
  • Correct numbering continues across pages.

I prefer not to store the entire response temporarily and display everything at once. Any ideas would be appreciated!
Thanks in advance.