Logic behind uploading a large document

[SomebodySysop]

@darcschnider

The idea is you set your size limit per chunk so that if data comes back over your limits you want it to chunk it into manageable chunks. But one important thing is to ensure that each chunk is seperated at a valid point so not in the middle of the message.

Exactly what we are discussing here: Using gpt-4 API to Semantically Chunk Documents