It’s true I’m currently only sending the current “Book Title, Chapter Title, and Section Title” in the context when generating content, but I have plans to add the ability to send
The entire document outline (Table of Contents), and/or
The entire chapter or section leading up to what you want go generate next.
Including 1 or both of those in the context each time. So even though I haven’t seen the duplicating issue myself (that you described), I bet doing this will solve the kinds of problems you’re talking about, at least for non-fiction writing.
here’s a generic approach (I think the book writing app is a killer use-case, if implemented right, I’m just busy with an edutech that’ll probably eat my next 10y of life, in a good way, so I’m giving away everything I know):
the more you zoom-in, the more condensed should the context that you carry over with you be. but this is key: always carry the whole context on your back, and compress only as much as needed to fit into the allowed window (32k tokens as of late, IIRC?)
example: you “zoom into” chapter 3 to write it, but make sure your back context (chapter 2 and 1) are also brought in and summarized just enough to fill the whole context length
pseudo-formally:
A = summary ch1, summary ch2, instructions ch3
B = general style, chapter 3 specific style, other generic styles
C = the response result
A+B → leads to → C
but, to maximize quality, make sure len(A) + len(B) + len(C) = 32k tokens (or whatever your max context length is)
the art consists here in choosing proper weights for A, B, C
applying these would make your app ready for the upcoming gigantic contexts, so you’ll just sit waiting for them, since your algorithm won’t hardcode context lengths, but percentages of a max_content_length input param
I had come to the same conclusions, but my purpose was to let token consumption be kept at a minimum.
I agree if you want to maximize both performance and cost, then presenting a summary of each chapter, rather than just sub-chapter section titles, each time some new content is generated, is a good idea. That is, aside from the [maybe already solved?] issue that the longer your context is the more the AI tends to forget parts of it.
Again it matters a lot what kind of book you’re writing. If it’s a novel, rather than non-fiction, I think that changes things a bit. Novels probably need longer context for any query, because those details can only come from the book itself and not from knowledge the LLM already contains inherently.
the theoretical promise of LLMs is they forget nothing from the context they claim to have. on the app side, sure, it’s a different story, based on how you’re implementing it.
if you’ve experienced gpt forgetting “important” stuff, while you kept the context within its claimed max length, that’s a bit worrying. sounds more like a bug to be reported to openai
Not to quibble over terminology but the word “forget” is a pretty broad term, because often LLMs can hallucinate, fail to follow instructions, and fail sometimes at the Needle in Haystack problem. It’s not bugs, it’s the nature of how even the state of the art current generation of LLMs work.
Anyway another reason not to just flood the context window just for the sake of flooding it, is that there are usage limits, rate limits, and financial budgets that people have, so I’m trying to achieve good cost/performance ratio.
I’ve developed a few fine-tune for my work and I found the operating cost to be pretty low and the price is going down by leap and bound all the time.
If a person is going to write a whole book, say 1,000 hours at least, the cost is not that big of deal. Say it’s 3,000$ over a year but if it saved 500 hours… that’s a hell of a return
It’s also going to likely soon be practical to just do a prompt like “Write a book on Python, for beginners.” and it will be able to complete that task without human intervention. So the question isn’t “can I fill the available the context window size allowed”, but it’s instead “let’s just put all relevant info in the context window, to get exactly the output we want”.
Yes, but at this point, just like in many ways it is now, you don’t need to book for Python for beginner anymore. You can just chat with GPT until you get the gist of it.
And then ask questions as your need arises.
But to write fiction and entertainment, for example you need to give it some meat. The same if you write on some arcane technical thing that it won’t have the information.
@wclayf Cool project, I’ve been thinking a lot about a project like this for fiction writing.
I think the ‘problem’ you are facing right now is that you are asking an LLM to create a book structure with word predicting technology that is trained on ‘the internet’. That is not what a good book is. I think having an agent act as a senior dev and another agent as a writer is a good start. But I think you’ll have to come up with much more of a theoretical framework to make a project like this really product commendable results.
For my fiction example: characters need to be relatable, believable and we need to ‘buy into’ their story. Story structure is fairly well understood. If you can figure out a way to make a multi agent framework come up with e.g. a character, a story arc according to the Heroes Journey, start with a skeleton first (more of a screenplay) and have another agent dress it up with flavor and details, now we’re talking. Its really about how to bring all those theoretical frameworks on genre, character, story and plot together in my opinion.
For non-fiction YMMV, but some ideas tho think about (I bet there are theoretical frameworks written on how to make a good educational book).
I agree. Creating just the book outline (which in Quanta can be done either manually or by AI) and then letting the AI fill in the content based on that hierarchy is just the first step.
As I mentioned above it can generate decent non-fiction, but for fiction writing (novels, stories, etc) we need to do more to make the AI understand the characters and the plot. Lots of authors would want to write the general plot themselves completely by hand (an outline of sorts itself) and then let the AI generate the actual narrative text between book characters for any given scene, given just the “goal” of what needs to “happen” in that scene, based on the plot.
That’s totally doable too. I’ve experimented with asking OpenAI to generate a narrative “In the style of Tom Clancy” between two characters, where I give their names, and explain only what needs to “happen” (at a high level, like in 3 sentences) and it can write what looks very much like several pages of a Clancy book.
I may never do this however, because I’m not too interested in fiction writing. If someone wanted to pay me for it, I would do it.
Hi there I’m trying to write a romance novel but soon found that the generated ouput was very low on word count - I requested 9000 words per chapter but usually only got 900 words ! So I’m very interested in using AI Story Builders which I have installed on my laptop but not quite sure how to start !Do I need to be using Azure ? I have a beefed up description of a story with detailed chapters - is this enough to get started ? ( no fine tuning yet ) Thanks
I’m convinced the following approach works best, where every prompt knows:
The entire outline of the book, in broad terms.
which chapter it’s writing in right now.
the outline of everything that will happen in that chapter
which part of the chapter outline it’s working on
what has already happened in the chapter, as a summary, so it has that context and won’t recover old ground.
the exact previous 100ish sentences of narrative, so that it can pick up at that exact point in the narrative, in a natural seamless way.
The general approach is when generating new text, you need ALL of the immediately preceding text (up to a limit), then as you zoom out at every level, you give more and more compressed summary-type information about the level it’s in.
Sort of like the context window has a focal length like a lens. The closer you are to the actual place you’re adding text, the more you need ALL of that context, but the further back in time, the more you can rely on summaries, and outlines. I hope that makes sense.
This is exactly what AIStoryBuilders is doing. Every step.
Yest, still I only trust the AI to write up to 1000 words. From nearly a year of working on this issue I believe: “Chat GPT is not smart and cannot write good prose”. What I also believe: “It can help write prose”. If anyone disagrees, I would love to see the output where they are able to get the AI to write more than 1000 words and it doesn’t suck
Hii jsbinette! Yes, I am deeply interested in you sharing your extensions with me and telling/guiding me about your experience with the development of the book. cajirdon@gmail.com
This is a very interesting thread. I wrote a non-fiction craft book about 40 years ago which was published and it’s now out of print. I’ve been thinking about republishing for sometime and I have experimented with chatgpt to edit some things I’ve written. My book is out of date the publisher no longer trades and gave me permission years ago to republish. My question is should I get AI to help me rewrite it? It will need lots of checking after but should I let AI loose on it to see if it can bring up to date? There are books on the market, particularly on Apple Books, and generally online, which haven’t been updated but are republished. Has anyone else had experience of editing with chatgpt? My main concern is it might be obvious it has been generated by AI.
