The Art of Fine-Tuning: How I Used GPT-3 to Bring My Podcast to Life on the Page

Community
1

LITOF Novelization Cover Mockup
LITOF Novelization Cover Mockup1024×1024 150 KB

I’ve always been drawn to the potential of using artificial intelligence to augment and enhance the creative process. When I started working on my podcast series “Lights in the Old Fort: A Symphonic Mystery Adventure” with my brother back in 2014, I never could have predicted the journey it would take me on.

The podcast was a grandiose, multi-layered production featuring music, sound effects, and over three dozen voice actors. It had all of the ingredients for a Hollywood blockbuster - except for one thing: an audience. Despite its high production value, the six-part series was released in 2018 to little fanfare or acclaim.

But I didn’t give up on my baby. This year, I decided to take on the challenge of turning “Lights in the Old Fort” into a novel. I started by converting the first 10 chapters of the podcast script line-by-line myself, but I quickly realized that this process would take an incredibly long time to complete for what more and more seemed like a vanity project.

That’s when I had a hypothesis: could I train GPT-3 to help me write the rest of the novel?

First I created a fine-tuned model using lines in the podcast script as prompts, and the equivalent lines of narration and dialogue from the novel as the completions.

The results were astounding: the machine produced passages that flowed smoothly from one scene to another and even introduced new characters convincingly. In other words, if you didn’t know better, you might think a human wrote them too - albeit with some help from an extremely smart computer program.

But the process of turning the AI’s output into a coherent novel was far from easy. Not only did it take months of fine-tuning and curating to get the output to a point where it could be incorporated into the book, but the costs were also significant. As a fine-tuned model, GPT-3 was vastly more expensive to use than the general models, and I estimate that I spent close to $2000 or more on processing costs alone.

But despite the challenges, I was addicted to the process. In every line, I felt like I was able to keep tweaking until I got the optimal version of the idea or emotion I was trying to convey. It was like shopping in a mall with infinite stores, each one with something that looks great and fits you, but the next store is just as likely to have something that might look even better. You’re just about guaranteed to win something with every pull of the handle, but there’s always a chance you’re one away from the jackpot. How do you know when you’re done?

The end result of this process is “Lights in the Old Fort: A Novel,” a novel that is the first and most extensive project of its kind. It’s a seamless blend of human and machine writing that I think is truly remarkable, and I hope you’ll all agree.

The novel is the my attempt to a capture a narrative as vivid and intricately woven as the podcast’s unique lush musical score. I can’t wait for you all to read the final product and see what you think.

I’m eager and terrified to see what the future holds and how technology like GPT-3 can continue to enhance and augment the creative process. It’s an exciting time to be a data scientist, and I can’t wait to see what the final product will look like once it’s been edited and revised. It’s been a long journey, but it’s been worth it to see the collaboration between human and machine writing come to life. I’m excited to share it with all of you and see what you think.

I’m grateful for the opportunity to have used GPT-3 so extensively and to be among the first to use the cutting-edge technology behind this project. It’s truly an exciting time to be a data scientist and see how AI can continue to enhance and augment the creative process. I can’t wait to see what the future holds and what the next generation of AI has in store for us.

So stay tuned for updates on the release of “Lights in the Old Fort,” It’s going to be a wild ride, and I can’t wait to share it with the world! https://brothercast.net/

5 Likes
2

Nice write-up. Thanks for sharing.

Welcome to the community. Hope you stick around.

2 Likes
3

Wow, I really like your tenacity and your result. Any chance you can share your process? I’m doing the same thing you just did, looking at GPT-3 and wondering if it would assist in finishing a novel I started (a vanity project). But I can’t seem to find: 1) what an appropriate fine-tuning dataset should look like for this type of project, and 2) how to train GPT-3 to use that dataset to become an assistant. Hoping you’re willing to share what you’ve learned.

Thanks,
Chas

4

Hey there! Thanks for your acknowledgement!

This project happened to be a perfect fine-tuning use case since I’d already completed the entire podcast script and about a third of the novel. One thing that worked out to be a huge advantage: because the podcast was essentially a radio drama, most of the lines were already written for the characters tell the story and describe the scenes through their dialogue, which kind of cues up the direction and flow of the narration.

It still takes a TON of curation, especially to accomplish in a way that is consistent with your thoughts on the end product. It was truly fun to see how GPT-3 embellished scenes or backstories of the characters that I know so well in my head.

At the same time, you pretty much have to have your hand on the till the whole time if you want it to go in a specific direction. I also found it most useful to limit the output to about 200 tokens at a time, which means a pretty narrow range that you can work on at any given time. Any more tokens than that and the completion tends to drift off into garbage.

It was also interesting to see how ChatGPT was helpful and how it wasn’t. While it can handle and think about a lot more text at a time, it doesn’t have access to my fine-tuned models, meaning it can’t automatically pick up on my voice. I was hoping I could just upload the whole novel and have it be able to give this omnipotent resource for feedback about the anything in story. Turns out there are definitely limits on ChatGPT’s attention span.

Where ChatGPT is really useful is being able to drop a block of text into it when it’s not quite what I was hoping for and tell it to punch it up in some specific way. I love being able to say, “In my head, these characters were dangling from a rope and sweating profusely. Can you update the passage to include this?” and have it spit out exactly that. It’s also interesting to go back to interacting with GPT-3 after using ChatGPT for a while. I keep finding myself trying to write chat-style prompts to GPT-3 which is a lot more of a crap-shoot as far as consistency, even in Edit mode which should ostensibly be a little more like the Instruct models ChatGPT was trained on…

As far as fine tuning goes, I created a spreadsheet that had chunks of lines from the script in the Prompts column and the equivalent novelized text in the Completions column. My hunch was that if I could train GPT-3 on the way that I would convert each of these lines, it might be able to extrapolate the rest of the story from the script as well in my voice. And I was delighted to find that it was able to do even better than I could have possibly hoped.

The instructions for fine-tuning are very straightforward if you’re at all familiar with Python, and if not, ChatGPT can tell you exactly what you’d need to do to prepare a file for fine tuning.

In general the way I use GPT-3 to write is to first write an outline of the document I’m trying to create and then plop it into the Playground to flesh it out with GPT-3 until it’s something I like, and then curate whatever results are produced into a finished product.

So that’s been the process in a nutshell. It’s by no means quick or cheap, I find the results are astounding. Let me know if you have any specific questions!

4 Likes
5

Great, thank you so much for your reply. I’m glad to say my newbie efforts weren’t too far off. I’m decent with python and have code in place that reads through novel PDFs, creates writing prompts based on outlines and completions based on chapter text. I then uploaded that dataset to OpenAi. I guess my next step is to read up on curation and testing - my understanding is: I interact with the new model with new prompts, capture those outputs that are good in complexity and length, create a new dataset with that capture, then fine-tune it again? Is that right?

1 Like
6

Well in my case, I did a sort of continual update of the fine tune for a while. Each time I completed a chapter I would append my spreadsheet with the latest script/prompt novel/completion pairs and train a new model in the hopes that it would glean additional context and richness from the collaborative product. So in other words, each fine tuned model I created was the culmination of everything in the script and the manuscript up to that point. And was certainly able to achieve what I set out to accomplish with that!

I fine-tuned several more chapters until the costs started getting too high for my tastes - The last model I trained cost close to $20. By that point the model I was using was already so good at crafting incredibly vivid narration; narration which so accurately captured the nuance of the characters quirks - meeting or exceeding the versions I had imagined them in my head - It started to seemed like the returns would start to diminish if I continued training after each chapter, especially since then actually using the fine tuned models afterwards is so pricey.

If I’d had unlimited budget, it would be fascinating to keep building the fine-tuned model to see how much more accurate it could get.

The quality of the results are so subjective anyway that it’s anyone’s guess, but do I wonder what the limit might be. Ostensibly at some point you could train a model with enough awareness to generate a whole book in this fashion with no human intervention.

But that’s not what I’m rooting for, and it’s not what I think will happen.

Really I wanted this book to be an example for others of how AI can actually augment, rather than replace humans in their creative pursuits. I wanted to show that AI can be used in a collaborative way, and ultimately produce something that was never possible before. It’s a cool time to be paying attention :slight_smile:

Do you have a specific aim with writing/engineering/curating your novel?

1 Like
7

That you for sharing this experience. Sounds so cool! What did your fine-tune dataset look like? I.e., how did you structure the prompt/response pairs?

8

Hello!

I created a spreadsheet that had chunks of lines from the script in the Prompts column and the equivalent novelized text in the Completions column. My hunch was that if I could train GPT-3 on the way that I would convert each of these lines, it might be able to extrapolate the rest of the story from the script as well in my voice. And I was delighted to find that it was able to do even better than I could have possibly hoped.

Podcast Script (Prompts):
ANTHONIO: … Hey hold up a minute, guys…. Dathaniel, look at the grass-- It’s been flattened down like a car drove up here…

DATHANIEL: You think those guys we read about in the note came up here last night? I don’t think these tracks were here yesterday…

ANTHONIO: Hmmm - it looks like there must be a road that goes off deeper into the hills….we must not have seen it yesterday ‘cause it’s so overgrown!

DATHANIEL: That’s strange, I wonder where it would lead to - There’s nothing north of here except miles of forest until the Canadian border!

Novelization (Completions):
“Hey hold up a minute, guys! Dathaniel, look at the grass; It’s been flattened down like a car drove up here!” he says, pointing to several long swathes of crushed vegetation.

Dathaniel kneels beside his brother and closely examines where Anthonio points. He brushes aside some tall blades with one hand while holding them steady with his other palm. The tracks are quite clear; they seem fresh as well… They had not been there yesterday for sure.

Anthonio rises again and looks around more carefully this time. He scans the clearing beyond what was visible from the bluff. A faint trail is barely discernible through dense undergrowth along an adjacent ridge which extends much deeper into forested hillside beyond Fort Bowden

“It looks like there must be a road that goes off deeper into the hills," observes Anthonio, “We must not have seen it yesterday 'cause it’s so overgrown!” “That’s strange,” remarks Dathaniel, “I wonder where it would lead to? There’s nothing north of here except miles of forest until the Canadian border!”

Surprisingly straightforward process! Are you working on anything similar?

9

That is so cool! Very clever idea, and that’s awesome how well it worked!

I’ve been wondering how to capture the voice of a particular style of writing. I’d been thinking to try something like prompt:, completion:, but I’m not sure if that’s a particularly fruitful approach…

1 Like
10

It’s a pretty low barrier to entry to find out! I’d say it’s worth exploring at least, and I imagine you’ll find your jaw on the floor :smiley:

1 Like