Using gpt-4 API to Semantically Chunk Documents

API
136

OK, after 2 months, I’ve got a fully functional system up and running in real time.

This is the process:

Hierarchal Chunking System (OpenAI)
Hierarchal Chunking System (OpenAI)1009×1557 93.8 KB

  1. export the pdf (or whatever) document to txt.
    1. I am set up to use: AWS Textract, PdfToText, Solr (Tika), PyMuPDF and Marker
  2. run code to prepend linenoxxxx:
  3. send this numbered file to model along with instructions to create JSON hierarchy file
  4. process the JSON file with code to:
    1. add end_line numbers
    2. add token_count totals for each element
  5. run code on modified JSON output to create the chunks.
    1. semantically sub-chunk chunks that are > x tokens
  6. add chunks to your embedding JSON to be uploaded to vector store.

Notes:

The weakest link in this system is the model API call to create the JSON hierarchy:

  1. the returned JSON file could exceed the 8K model output token limit
  2. model sometimes gets creative and doesn’t return strict JSON

The actual language of your hierarchal chunk prompt will change depending upon your document types. This is to be expected. The semantic chunk prompt I am using, however, appears to work in most cases.

Needless to say, because these are working for me doesn’t mean they will work for you. You will need to modify as appropriate for your use cases.

After almost 2 years of resisting the use of Python, I finally gave in in order to be able to take advantage of the Marker and PyMuPDF markdown extractors. I run all the Python code in Docker containers.

I also needed to modify my system to allow for queued processing of text extractions as Marker markdown can take anywhere from a few minutes to over an hour.

I have discussed why I believe this process is superior to most existing methodologies here: Using gpt-4 API to Semantically Chunk Documents - #112 by SomebodySysop

This could very well change in the future, but for now, I’m pleased with the results.

I’ve already posted several examples of test inputs and outputs in this thread. Moving forward, I will continue to post examples. It is currently installed as part of my embedding pipeline, so I will get a very good picture of what works and what doesn’t work.

Many thanks to all who have contributed to this discussion, which as been a tremendous help in getting to this point.

3 Likes