I’ve been using the API to write books. I’ve been having the AI summarize what it has written so far, make a note of where it left off, and then continue writing. The results are decent.
There are 2 main problems I am noticing:
Consistency: A single sentence 5 chapters ago might have an effect on something being said later. If a character is afraid of blue hats, but he buys a blue hat later with no explanation, that is an inconsistency.
Repetition: The AI will rewrite the same events at different parts in the story. It has forgotten the conversation from many chapters ago, so I can’t blame it for not knowing it has already done something.
I’ve done various things to try to mitigate these issues with varying success. I wanted to try training or fine tuning a model on each chapter as it writes them. I wonder if this would improve (or maybe make worse) the problems of consistency and repetition. I’m asking because I wonder if would it be a waste of time and money (like this whole endeavor is anyways) or if it might actually make its writing better.
I can’t speak to fine tuning or training - but I wonder what other options you have…
Have you explored anything like:
Multi-step “pipeline” for writing? Maybe you have it write a chapter, but then ask it to evaluate for consistency - chapter by chapter (I’m not sure how much text a chapter is, hah).
Maybe you ground the characters, plot, etc. in an easy to consume format to reference during the writing of each chapter. I’m thinking about something like modeling/noting the significant attributes of a character in JSON or something succinct but clear.
(adding on) Can you record the high-level plot/events in a succinct format such that you can provide this deeply “historical” context to later chapters?
Like I said, I’m not well-versed in fine-tuning or training but I thought I’d throw out some other ideas
I have implemented some of these things. The characters are all referenced during writing, with short bios, and updated when necessary. There are summaries and chapter summaries being referenced. Maybe it could be better but I’m not planning to show my code. JSON is a good idea, but I’m not sure it would actually be better than a simple bio.
The multi-step pipeline is an interesting idea. My chapter sizes can be quite large. If there was a way to do multiple passes over a text to search for inconsistencies, and improve it incrementally, I’m not sure if a generative LLM is the right kind of AI to accomplish this. How would it know that something represented an inconsistency from 20 pages ago without having the entire book in its context?
The problem is that inconsistencies can pop up over non-major plot points. If you made a human writer write a book, page by page, without going back, they would probably make the same errors. I am curious if training can make its understanding of the world it has created be more rich, perhaps making it seem lived-in and real.
At the end of the day, it’s just going to be AI slop, but it would be interesting to see if it can improve.
My conclusion is that this is a bad idea for a couple of reasons.
Training data fed into the AI is supposed to be data that you confirmed as desirable. You should at least manually parse the results rather than blindly send everything back into the model. Otherwise, you reinforce bad responses.
Training doesn’t seem to help the AI remember fine details about the story. It can still mix things up, i.e. a character that was a blacksmith could become a tanner or something.
Training makes repetition worse. The more you train a model, the more it will simply repeat what it said before.
