I’ve developed a publishing tool that handles large text files, utilizing the GPT API for both editing and translation tasks. The system is generally functioning smoothly, barring a significant problem: the API operates at a painfully slow pace. For instance, processing a book with 500 pages and 90,000 words takes approximately four hours. I’m wondering if this duration is expected or if there might be an issue at play. Additionally, I’m seeking advice on how to expedite the process. Would increasing traffic on GPT-4 further exacerbate the speed issue?
2 Likes
So, you’re processing roughly 120,000 tokens in roughly 14,400 seconds, for a rate of about 8.3 t/s or 500 TPM (assuming the output translation is roughly the same number of tokens as the input text—more on that later).
That would seem perfectly reasonable for GPT-4.
It’s been pretty well documented though that most non-English languages require more tokens, so 120k input tokens could easily be 150k–500k output tokens, which would make the speeds you’re reporting all the more impressive.
So, I don’t think it’s fair to say it’s slow as much as you just wish it were faster.
You could always speed it up by splitting up your text and dispatching simultaneous requests.
If you are translating to a language which has a token density twice that if your input langue, the total number of tokens you’ll be using is 3x your token count or 360k tokens.
The API has a rate limit of 350k TPM so in theory you could cut this to a little more than a minute total.
If a single thread is capable of producing 500 TPM and you need to produce 240K tokens, you’d want to use ~ 500 threads sending ~250 tokens in each request.
To guard against hitting the rate limit and getting errors you’d probably want to space out sending each request by ~50ms± and the whole thing not including the overhead of splitting up the text would take about 90s.
1 Like
Thank you for that response and calculation. When I say slow it is a end user perspective or his expectation. Maybe I have to change the UI. A user does not want to wath streaming like chatGPT for 4 hours. The problem with multi threading is the error handling and I still hope to get a summary across all posts. It is not documented but chatGPT-4 keeps track of all posts in a session. I know the api is stateless.
I think your users may need to moderate their expectations.
How long does it typically take to create a high-quality transition of an entire book?
One thing you might consider is starting out by breaking the text up into many small chunks, send the first x% of the text in parallel to the API (say 10%), start streaming the first chunk right away, add the rest of the first 10% as they finish, then reduce it back to single threaded (add a progress bar).
That way they can start reading right away if they’re impatient and they have a good idea of how long it will take to complete the translation. I’d also offer to let them navigate away and send them a link to the finished translation once it is complete.
Now, all that said, you need to be very careful here. If your users do not own the rights to the material they are submitting for translation that you will almost certainly be on the hook for facilitating copyright infringement unless you have absolutely ironclad agreements indemnifying you.
5 Likes
How long would it take a human editor to get the text back?
I would have them start the process and send them an email when it’s done.
4+ hours is better than 4+ weeks for most authors…
4 Likes
I have implemented a mercure server and display the chapters of the book one after the other once one is created with a typewriter effect.
Extending the timeline in between two plots can be done in parallel (don’t know how you do it though).
Anyways using this websocket technique the user can watch the book being written.
In this example text I just use a langchain technique like (give me a synopsis of indiana jones … rewrite characters, location and timeline and give it to me in the following format:…
I have way better prompts to get good results for books.
e.g.
Chapter 1: The Morning After
I woke up with a throbbing headache, feeling the soft sheets tangled around my body. For a moment, I couldn’t remember where I was, but as I slowly opened my eyes, the blurry memories from last night came flooding back. The room was dimly lit by the morning sun peeking through the curtains, and I could see the stranger’s clothes scattered on the floor. I looked around, taking in the unfamiliar surroundings.
The walls were adorned with posters of bands and movies, and a bookshelf filled with novels and knickknacks stood in one corner. A guitar leaned against the wall next to a small desk covered with papers and a laptop. It was a typical guy’s room, I guessed. My gaze settled on the man sleeping next to me, his face peaceful in the early morning light. Brad, I remembered his name.
I’m Sarah, by the way. I’m not the brightest crayon in the box, if you know what I mean, but I get by. I work at a gas station, which ain’t exactly glamorous, but it pays the bills… most of the time. My boss is a real pain, always yelling at me for being late or messing something up. I swear, I can’t do anything right in his eyes. And don’t get me started on my parents. They can’t stand me, and the feeling’s mutual.
As I looked at Brad, I couldn’t help but notice his strong jawline and the way his dark hair fell over his forehead. He looked so different from the guys I usually hung out with, and that’s probably what caught my eye last night. He seemed… I don’t know, more mature, more confident. It was refreshing.
I carefully slid out of bed, trying not to wake him, and picked up my clothes from the floor. I looked at myself in the mirror, wincing at the mess that was my hair. The night before had been wild, but that was how I liked it. As I got dressed, I thought about what would happen next. Would Brad be like all the other guys I’d been with, just another one-night stand? Or was there something more to this connection we’d made?
I tiptoed back to the bed and gently shook Brad awake. His eyes fluttered open, and for a moment, he looked just as confused as I had been earlier. But then he smiled at me, and I felt a warmth in my chest that I hadn’t felt in a long time. Maybe, just maybe, things could be different this time.
Chapter 2: A Hasty Goodbye
As I woke up to Sarah gently shaking me, I squinted against the morning light and took in the sight of her. She was smaller than I remembered from last night, and I couldn’t help but think that I might have had one too many drinks. But hey, I thought to myself, she’s still better than spending another night alone.
It took me a second to remember that my girlfriend, Lisa, was supposed to be coming back soon. We’d had a stupid fight before she left for her business trip, and I wasn’t exactly proud of how I’d spent my time while she was gone. I knew I had to find a way to get Sarah out of here before Lisa came home.
“Hey, Sarah,” I said, rubbing my eyes and sitting up. “Last night was fun, but I’ve got a lot to do today. Would you mind if we, uh, call it a morning?”
She looked a little hurt, but she quickly masked it with a smile. “Sure, no problem. I’ve got to get to work anyway. Thanks for the, um, great time.”
I felt a pang of guilt, knowing that I was brushing her off, but I didn’t have much of a choice. Lisa could be back any minute, and the last thing I wanted was to cause another fight.
After Sarah left, I hurriedly cleaned up the remnants of our night together. I couldn’t stop thinking about her, but I had to focus on making things right with Lisa. I sent her a text, apologizing for our fight and telling her how much I missed her.
I waited for a reply that never came.
Feeling a little anxious, I decided to take my mind off things by doing some chores. I went to the store to buy some groceries, hoping that Lisa would appreciate coming home to a well-stocked fridge. I picked up some of her favorite snacks and ingredients for dinner, hoping that it would help us patch things up.
After putting away the groceries, I headed to work. I’d been learning about machine learning for the past couple of years, and I recently started working on a project that involved developing AI-generated voice technology for automatic sales calls. It was a challenging task, but I enjoyed the mental stimulation, especially as it helped me escape from the worries of my personal life.
As the day went on, I found it harder and harder to concentrate on my work. I kept glancing at my phone, waiting for Lisa’s response. It felt like there was a knot in my stomach, growing tighter with each passing hour.
Chapter 3: Sarah’s Obsession
As I stepped out of Brad’s apartment, I couldn’t help but feel a strange mix of emotions. I was hurt that he’d brushed me off so easily, but at the same time, I was drawn to him in a way I couldn’t explain. As I walked down the hallway, I made a sudden decision. I wasn’t going back to work. I wasn’t going back to my dead-end job or my miserable life. I was going to follow Brad, find out everything about him, and make him realize that we were meant to be together.
I quickly ducked into an alcove near the entrance of the building, waiting for Brad to leave. I felt a thrill of excitement as I crouched there, hidden from view, the anticipation building as I watched the minutes tick by on my phone.
Finally, I saw Brad emerge from the building, looking preoccupied as he locked the door behind him. I waited until he was a safe distance away before slipping out of my hiding spot and following him. I tried to stay inconspicuous, keeping my head down and maintaining a casual pace. I felt my heart pounding in my chest, but I couldn’t help but feel alive for the first time in a long time.
I followed him to the grocery store, watching as he carefully selected items and placed them in his cart. I couldn’t help but imagine us shopping together, planning meals and laughing as we navigated the crowded aisles. I felt a growing obsession, and I knew that I needed to be a part of his life.
When Brad left the store, I didn’t follow him to his office. Instead, I waited until he drove away and then returned to his apartment building. I knew it was a risky move, but I was determined to get closer to him. I found a window that had been left slightly ajar and used it to break into his apartment.
Once inside, I looked around at his belongings, trying to piece together more about his life. I found his clothes, his favorite books, and even his toothbrush. I felt a thrill at being so close to him, and I couldn’t wait for him to come home and find me there.
As the hours dragged on, I cooked dinner, hoping that the smell would be enticing when he returned. I knew that I was crossing a line, but I couldn’t help myself. There was something about Brad that I couldn’t resist, something that made me want to throw caution to the wind and do whatever it took to be a part of his life.
…
Do you send the requests 1 at a time? You can try sending multiple at a time to improve the performance, up to a limit, of course.
For a novel that is not possible of course. Chapters building up upon eachother.
I create a universe first where all happens, then characters, locations where the story takes place and a short outline.
Then I prompt deeper between or at locations.
e.g. create a story about how the character talks to companions or himself while traveling.
Describing the way… it’s pretty complex and requires a graph data structure that stores everything that has been told.
Some characters can’t know about stuff and some do. Especially crime genre is complicated.
I mean you have to prompt pretty abstract and have a variety of prompts to not make every book be the same.
Well, to finish it I need to wait until I get access to 32k context. Quality over speed.
Meanwhile I got access to plugins on May, 14th and built a jobsearch plugin which i submitted yesterday.
There may be some fine tuning possible. Out of interest, which model are you using and how are you breaking up the task (to get around the token limit)?
An easy solution for that is to keep context in an abstract way and then extend between two time line points with a sub time line…
Mine is a little bit more complex.
Jochenschultz, when you say “keep context in an abstract way and then extend between two time line points with a sub time line” can you give an example of this?
I can explain it abstractly
When you create a new chapter you can add some context to the prompt that generates it.
From your outline you already know who is going to play in that chapter so you extract that information e.g. the names Jane and John and load everything you know about them from your data storage and add that to the prompt that creates the chapter.
This way you don’t have character with three types of hair color etc…
1 Like
But there is way more information than just hair color.
I give you an example:
Two people (John and Jane) meet in Chapter 1.
They go different paths.
John goes to the supermarket and meets Tom.
Jane goes to her office.
Tom tells John that Jane’s mother died.
Jane in the next chapter starts crying because her mother just died (but she couldn’t know).
So you need to keep track on who said what to whom and who went where and who knows about that.
For that I use different agents, a graphdb and spatial techniques.
Waiting for GPT-4 32k model access atm before I can launch (and of course there is a system for proofreaders so the market is not flooded with stupid books).
How do you do it ? I did try chunking and sending but GPT lose its context for large payloads. Any help on API to send the chunks would be helpful.