New approach to summarize books

[SomebodySysop]

@jr.2509

I also consider it more intuitive to break up documents by section as opposed to fix chunks that don’t take the document’s logic and structure into account as it can affect the quality of the summary.

Aha! Semantic Chunking! https://youtu.be/w_veb816Asg?si=fKodYwnF1rupHr0V

@jr.2509

To begin with, it’s a use case specific consideration. The documents I am applying summarization on may be different from yours.

Absolutely. Not only case by case, but sometimes document by document. I have multiple custom embedding schemes within one dataset because different documents sometimes require different embedding methods: sometimes I add a summary of the entire document to each chunk, sometimes I summarize each individual chunk, sometimes I just place a overall description in each chunk, and sometimes I don’t add anything because the title and other metadata suffices.