Ways to automate breaking a large piece of input into chunks that fit in the 4096 token constraint?

[aikanarov]

You can use many techniques like map reduce, refine, etc… I use langchain for it.