A theoretical novel writing tool

General API discussion
5

I guess a different approach would be alternating between generation and summarization to keep the AI on topic.

GENERATE:

Use a prompt starting with:

  • Overall summary - The story so far
  • Medium Term Summary - A summary of recent events
  • Short Term Summary - A summary of the last completion

It gets harder as you move forward if you don’t want to blow through the token limitations.

You’d have to set budgets for prompts/completions I guess the limit for davinci is what 2048 tokens right?

For example:
500 tokens for The overall summary
150 tokens for the Medium Term (optional)
398 tokens for Short Term summary
200 tokens for any structural wrapping needed (e.g. character list, setting description)
800 tokens for completion (that should give you 2 pages of a book? maybe?)

This budget will likely have to be a little flexible because "The overall summary will grow as you progress, so I’d keep a buffer for it, but I think in the early stages you could get away with longer completions.

Store the full completion as book pages .

SUMMARIZATION

  1. Summarize the returned completion add to “Short Term Summary”
  2. Summarize the “Short Term Summary” Add to the bottom of the Medium Term Summary (remove the top), and to the overall summary.

This would be a fun and relatively quick python script to write.

5 Likes
6

Fascinating. I’d love to play around with such a tool if you were to code it. Definitely let me know how that goes.

One thing I’m trying to be mindful of is using GPT-3 to write things the way they’re “supposed” to be. I think of it kind of like playing an electric guitar plugged into an amp after playing an acoustic guitar. Screeches of feedback are technically “wrong” but guitarists quickly incorporated it into their playing on purpose. I actually like the things that GPT-3 does that seem “off” or “incorrect.” The form of the novel is already amazingly diverse, from realist 19th century novels, through the stream of consciousness experiments of modernism, postmodern weirdness ala Pynchon, magical realism, and now the almost feed-like work of younger writers like Patricia Lockwood.

I guess the thing I’m looking to do is create an ongoing repository of chapters so that the AI can at least be aware of them, and it would be ideal to not have to summarize “our story so far,” although I can already see how that actually might be fun. The structure of the novel as we know it is still largely beholden to Charles Dickens, who serialized his work in newspapers, the cutting edge technology of his day. All of this is to say, I’m looking at GPT-3 to blaze a trail into the future of literature rather than replicate what has come before. Glad to connect with other minds puzzling over the same questions.

2 Likes
7

I started doing some investigation this morning for fun, the rabbit hole gets deep quick.

This is all in the playground, before starting on the scripting tool, just to gather requirements for the code.

BACK COVER BLURB
I started with generating a Back-Cover-Blurb for a book, with a prompt and without a prompt. this step was easy and works either way. (davinci-instruct-beta was really good at the task).

FIRST PAGE
From the back cover blurb, I started on the first page. I could have simply started from scratch, but I wanted to see how much I could stick to what was in the blurb (this will be important for keeping track of the book as a whole without being prescriptive of what happens.

The first page differs from the following pages as it doesn’t have anything other than the back-blurb, to start from, so I started with

BACK-COVER-BLURB: +[whatever I generated previously]+CHAPTER 1:

as prompt, using davinci, returning 400 tokens to get about a page of a book.

If the page ends in a period. I assume it’s the end of a paragraph. and store that entire page.

If the page ends in an incomplete paragraph, I trim it and store to use as part of the prompt of the next page.

SUMMARIZING THE PAGE (this is a function that will get repeated with every page generation)
The next step was to Summarize the first page output. This needs to be done with a lower temperature, I tried instruct, but the best results I got was using davinci with a prompt like this:

After reading the following text, the teacher asked the student to summarize it.

  • [PAGE CONTENTS] +
    The student provided the following summary:

SUBSEQUENT PAGES
My plan here is to loop through this step 200 times, this will most likely not lead to any recognizable narrative arc, but it will spit out 200 pages.

  • The prompt will contains:
  • The back-cover-blub
  • The recent events summary
  • The last complete OR incomplete paragraph (as start)

FINDINGS

I feel like I will have to run grammar correction steps after every generation as I have noticed that the engine can start dropping the ball if your text contains grammatical or spelling mistakes, it will incorporate it into its styling.

If your frequency penalty is high the engine will start penalizing punctuation from occurring in the later parts of the text. I don’t quite know how to stop that from happening other than keeping the generations short.

Is there a mechanism where I can exclude tokens from being penalized? I think I’m going to open a thread on this.

3 Likes
8

Fascinating. I didn’t realize that about the high frequency penalty penalizing punctuation. I’ve noticed that a number of my completions end with strings of words that fractal out into word salad. I first started to just lop these off and discard them, but lately I’ve been sifting through those parts of the completion looking for interesting word combinations that I can then incorporate into the next iteration of the prompt.

1 Like
9

Good thread. I’ve been too busy with other stuff to read a lot here, but thanks for this… I’ve been experimenting with a few things over the last year… everything from children’s stories (chapter books) to never ending narratives based off character backstories generated by GPT-3 at LitRPG Adventures…

One of the things on my todo list is to fine-tune for blurb generation… and some other stuff… oh for more time and money!

ETA: I’d love a bigger context window in the next version…

3 Likes
10

@PeculiarPath I believe that with good prompt design and preparatory work (I like your idea), the current version of the OpenAI interface might suffice to write ‘a novel’ (I think of approaches like the snowflake method), but the lack of an over-arcing ‘log’ (as @boudinot calls it - the actual writer’s “memory”), will almost certainly result in less interesting fiction (and almost certainly not in “literature”).

1 Like
11

There are several methods out there for fiction writing, I like the structured approach (regardless of what method) because the groundwork of codifying the requirements has already been done for the most part.

Choose a method, turn it into an algorithm. Of course that, like with any creative work, a lot of the “formulas” out there contain a great deal of subjectivity, ambiguity and are very often vague and open to interpretation. Even with Larry Brooks’ story engineering, and story physics with titles that suggest a more exact approach, creativity is hardly an exact science, and humans cannot be beaten when it comes to that.

The processes I describe there are mere attempts to use those methods to deal with the limitations of the System (memory, planning, plotting) in order to mimic the structure present in those books.

My crude understanding of the nuts and bolts behind GPT-3 is that it is a highly trained prediction engine that calculates what is the most probable word (token actually) that will come next, base on what has come before. with the definition of “Before” being your immediate, limited, prompt, and the whole of the corpus used for the pre-training, which generated the 175B parameters that make davinci.

Predicting the next token is hardly planning ahead, let alone being able to set up things to pay off in later chapters, keep track of character traits and personality, to know how a character is likely to behave.

12

I keep going back to the analogy of synthesizers and drum machines. When I started using GPT-3 I started listening to Nine Inch Nails again. No one could ever accuse Trent Reznor of making music that wasn’t emotional or expressed some sort of fundamentally human spirit.

In my opinion, if you look to GPT-3 and AI in general as a means of replacing the process of writing, then you’ll end up disappointed. Writing to me is a process that can never and maybe should never be completely understood to the point that it can be reverse engineered. It exists in the same category as dreaming and sex, an experiential rather than theoretical activity. Susan Sontag in Against Interpretation said “In place of a hermeneutics we need an erotics of art.” My understanding of this is that reducing the writing process to merely an algorithm negates the very reason it exists in the first place, as something to be experienced subjectively, emotionally, viscerally.

It’s the collision of algorithms and the subjective nature of art that excites me most. I love all this thinking around how to bend the tool to produce a certain kind of output, while recognizing that it allows for the creation of entirely new art forms and genres, like electronic music, that wouldn’t exist otherwise.

1 Like
13

Kinda had the same idea when I saw all those articles about GPT-3 and journalists using the AI to write their paper.
While waiting to be granted permission, I tried experimenting with GPT-2. With the last version, I was able to feed my novels and short stories to the AI (because, also, I’m in Québec and I write in French, so I had no idea how it would react). I wanted to see if an AI could write with my style.
Could it be used to write a short story juste like it did with the articles ?
Short answer : no.
Long answer : hell no !
Even though I could recognise names or words from my novels, the phrases made no sense (at all!). I would basically have to rewrite everything myself, which is not the point.
GPT-3 gave me more hope for about a second. The writing is better, but it goes nowhere, says nothing, doesn’t have substance.
The thing is, part of the stories we write isn’t even on paper. It’s in between the lines.
Plus the AI works with rules. You teach it phrase or story structure, or grammar, and such, and it will reproduce that. But artist are constantly breaking the rules. Can you write a rule for the AI to break the rules and be original ?
I have to agree with @Romolo. You can get lots of words, but it won’t be litterature. Not yet.

14

Literature is a slippery concept- it contains a judgement, and fiction written by AI has no pretentiousness.

However, I’m more hopeful than you, @py.villeneuve! The creative powers of GPT-3 for fiction writing are insane. It can lead to literature, but not in a push-the-magic-button way. A writer is needed, a writer who understand how fiction works ànd how prompts work, a writer who controls the rodeo bull. Any non-writer will be tossed off immediately.

I believe AI-enriched fiction will be a thing in a few years to come. It will produce novels that are literary and revolutionary. They will be written by human and AI together.

15

I think it comes down to whether or not we consider GPT-3 as a “labor saving device.” If so, then it’s going to disappoint us by not autonomously producing works of coherent literature. But if we think of it as an instrument that requires the same amount–but a different kind–of work from the writer, then I’m with @Romolo in feeling it can spark a whole new kind of fiction.

16

Dali could be used to provide a graphical overlay for the novel using heat maps for comparative analysis alongside thousands of other successful books.

17

I am not anticipating that any one model will do all of the work of story writing. I see potential for a set of task specific fine-tuned models to help with stuff like: paraphrase this / finish the scene / or something like make an analogy of highlighted text.

I’ve found that a “reverse summarizer” works very well for expanding on a block of text, which can then be expanded on again. Now I’m fiddling with controlled generation that has something equivalent to the “log” you mentioned, basically if you create a fine tuned model that is trained on all sections of story, if there is 1 tying element that connects the sections it keeps the output coherent.

1 Like
18

I’m super curious about your reverse summarizer and controlled generation. Can you share any examples?

19

Yeah sure, there’s three of them. First was is just the reversed summarizer, other’s are the attempts at controlled generation. The third ones… Interpretive lol.

3 Likes
20

Hey everyone! New to the chat! On the topic of Transformers being able to write a cohesive “long term” novel, Transformers must have some sort of differentiable memory attached.

Another issue is recurrence slowing things down. Ie when you train GPTm (m for memory I made it up!), you don’t want to individually input the output of the old prediction into the next training step - ie you don’t want any correlation between training step 1 and N. In GPTx/BERT/variants, they just assume the output is right (teacher forcing).

So the issues are:

  1. How to add differentiable memory to GPT?
  2. How to do this without a recurrence relation?

I first thought of something boring (haven’t tested). You make a matrix M (called memory) of size (m, f). f is the original embedding dimension (like 768). m can be massive say 20,000. For every batch of text, you pass the MH attention layers, dense layers all the way to the final softmax layer, then somehow copy BERT’s CLS approach and “extract” the CLS 768 dim vector and perform v=M*(CLS) which will get u a tall skinny vector (20,000 by 1).

Then, perform a long tailed sigmoid ie 1/(1+exp(-0.5v)) onto v. Then element wise multiply the sigmoid output with v^T. You’ll get a (20,000 by 768) matrix the size of M.

Then M(t+1) = M + 1/(1+exp(-0.5v)) * v^T. Then append M onto X (which can be very problematic), or somehow “summarise” M (ie say via a Clarkson Woodruff Transform shrinking M(20000,768) to say (500,768). You can even train the summarisation weight matrix S so we get:

M(t+1) = M + 1/(1+exp(-0.5v)) * v^T
X(t+1) = concat[ X(t+1) , S * M(t+1) ]

The CW Transform will just “freeze” S as a hash table.

This has 2 benefits:

  1. Incorporates long term attention. Ie the dot product makes similar memories remember even more often, and discounts not so important memories.
  2. Fixes catastrophic forgetting. The use of a long tailed sigmoid allows long term memories to stay inplace and not vanish.

However there are is a clear issue with this approach:

  1. Recurrence comes back! Batches must now be sequential… Ie previously u can have 1,000,000 books scramble each page, and GPT would be fine. Now GPTm needs to train ONLY on book 10 then 103 then 12039 with page orderings intact.
2 Likes
21

On further thoughts:

  1. A long tailed sigmoid acctually might be bad, rather the rank one update possibly takes care of catastrophic forgetting since at the start, the CLS token updates every line of the memory matrix M equally. Rather tanh might be better.
  2. A new [MEM] token can be added in tandem with [CLS]. During Masked Self Attention, instead of using an upper triangular mask of -inf and lower tri of 0s, add an extra [MEM] token which allows the whole sentence to attend to it. Then feed the [MEM] token back for every NEW page in a novel. The [MEM] token ISNT STATIC, and every new book wipes [MEM]'s state back to all 0s.
  3. CW Transform is not needed. Rather shrink the memory matrix down to 10 rows.
  4. Recurrence isn’t that bad! Each batch row will be a separate row of random novels (novel A, G, Z, K etc). Each new training step will be a new page for every random novel (novel A1->2, G1233->1234, Z24->25, K213->214). The recurrence relation is only every page / sentence.
  5. Each novel has it’s own memory matrix M intialised as all 0s. To not cause bias to creep into the model (say a novel has 10,000 pages), you train GPTm in windows of pages, then save the per book M to disk and get a new novel. Retrieve the book’s unique M later. Likewise every new batch might reselect novel X. Delete the old memory matrix M and restart training.
1 Like
22

Daniel,

I will first of all say that I don’t understand most of what you’re saying here, but that isn’t a criticism. (Or rather, it’s a criticsm of myself) Thank you for thinking about this and imagining what’s possible. I appreciate it, and am excited about where this might lead.

I am coming to GPT-3 as a creative writer rather than as a software developer. I can speak to the weird creative process of writing a novel as a human. I’m hoping to articulate what I envision for an AI writing tool to the point of someone like yourself being able to engineer it.

I currently have the first 35 pages of a novel, totalling about 7,000 words. I would like to find some way to use this as a prompt.

I am currently developing a way to write short stories with GPT-3. They typically start with a one-shot prompt. I provide a paragraph or two of contextual orientation for the AI, followed by the first couple paragraphs. Once I generate a completion, I edit it to my satisfaction or sometimes delete it altogether and start again. I play with the settings along the way, particularly the temperature. I keep resubmitting the growing completion until I run out of tokens. As I near my token limit, I attempt to “land the plane” so to speak, bringing the story to what feels like a natural conclusion.

I’d basically like to engage in this same process but with bigger prompts.

Does this process sound compatible with the system you describe?

Again, thank you so much.

2 Likes
23

No problems sorry on my part for being vague and I didn’t explain it too much!

Ye so ur method sounds reasonable. GPT3 handles a can’t remember a window size of X tokens (2,048 tokens?) Essentially around 1,024 words since tokens aren’t words but word pieces. So whole paragraphs are possible.

Say we have GPT0 (fake super weak GPT with 4 words as the window). I start the sentence as “Hello my name is”

  1. Input [Hello], [my], [name], [is] into GPT0.
  2. GPT0 predicts [Daniel] as the next word.
  3. Re-feed GPT0 with [my], [name], [is], [Daniel] into GPT0.
  4. GPT0 predicts [and] as the the next word.
  5. Re-feed GPT0 with [name], [is], [Daniel], [and] into GPT0.
    and so on.

So yes your current method makes sense. The issue now is the window size. Clearly a window size of say infinite size is not feasible. GPT-5 could have a window size of 2^16 (65,536) tokens or something (GPT4 seems to be just more efficient GPT3). Another option is to allow any window size up to the input size. This is possible, just implementation wise for batches sounds complicated. The optimization algos also need to be edited. This can allow “infinite” sequences.

Another option is as I mentioned a Memory Matrix. Essentially instead of GPTx forgetting ur previous input, it keeps a “running summary” of ur input.

Say we have GPTm with 4 tokens and a memory of 2 sequences and we start as “Hello my name is”.

  1. Input [Hello], [my], [name], [is] AND Memory = [0],[0].
  2. GPT0 predicts [Daniel] as the next word. Update Memory = [1],[2].
  3. Re-feed GPT0 with [my], [name], [is], [Daniel] AND Memory = [1],[2].
  4. GPT0 predicts [and] as the the next word. Update Memory = [51],[22].
  5. Re-feed GPT0 with [name], [is], [Daniel], [and] AND Memory = [21],[224].
    and so on.

Now GPTm “remembers” the long term context of ur novel. It can somehow remember 10,000 sentences ago the introduction, the plot lines, the characters etc inside the memory matrix.

24

Ryan, if you’d like to try to use your 7000 words as a prompt, I can set you up with some of my tools.

1 Like