Weird response on larger text chunks

I’ve been utilizing the CreateCompletionStream() function on the API, specifically with ChatGpt3_5Turbo, for automated text processing. The software had been performing brilliantly until approximately four days ago when the response took a bizarre turn, almost seeming like it was hallucinating. Initially, I suspected a software bug and commenced debugging. However, after an exhaustive two-day search, I failed to identify any bugs in the software. In light of this, I decided to revert back to a basic API test. I discovered that the responses returned to normal when I used smaller text chunks. However, when I increased the token size to roughly 900 tokens (though the model allows up to 4096), the responses once again became erratic and seemingly hallucinatory. Notably, there were no alterations made to parameters such as temperature or penalty. Has anyone else encountered a similar issue or have any insights into this?

2 Likes

Is this only while streaming? Also have you tried the larger prompt in the Playground?

Thank you, Foxabilo. I didn’t consider that aspect. I just tested it myself, and unfortunately, I encountered the same issue. The behavior of the language model seems erratic in this case. It’s important for OpenAI to address this promptly since encountering errors with just 900 tokens shouldn’t be expected. It can be quite frustrating when you have a fully functioning application and suddenly experience unexpected behavior due to code changes. It’s a surprising and unfortunate situation that can be difficult to comprehend without a clear understanding of the underlying changes made by OpenAI.

1 Like

Do you have some examples where you can show the behavior? Perhaps a Question and Answer session where you have expected behaviour below 900 tokens of prompt and erratic behaviour above 900?

You see the response does not make much sense. When I reduce the text size then it will response correctly.

ChatMessage.FromSystem - “Edit and proofread. Do not add comments.Style: Narrative. Audience: General audience. Don’t remove quotation marks.”

ChatMessage.FromUser - "Rajna saw herself from the outside, and it was indeed her own body because she could feel her disgust, touch herself, stare at herself incredulously. Naked, she sat on a cot in the white room where she had to taste the pain. It smelled like disinfectant.\nIn the brown corduroy pants over the chair, a pager had beeped. Shortly afterward, the parquet floor trembled under the weight of his quick steps. Without a word, he had left and closed the door behind him. The key turned energetically in the lock.\nAs soon as he was gone, the young woman fell back annoyed onto the examination table and stared desperately up at the ceiling.\nShe gagged. Saliva filled her mouth.\nHer skin was reddened and sweaty, as if she had been lying in a sauna after an infusion. She was so close to her goal, but now…she should have massacred him right away.\nHer body had become the battlefield of her senses, not searching for love, but diligently collecting the nectar in her brain greedily, from which our bodies boil with angry obsession. When memories of past thoughts touched her lips, when her vagina was filled, when sweaty bodies slid over each other, when the memories whipped her up, then the desire for his death surged in her mind. It was comparable to the effect of ecstasy; that intense euphoria, that high feeling that turns virgins into warriors, dispels any fatigue, forgives any hunger.\nRajna lay naked on the narrow bed with her arms and legs outstretched. Her limbs hung down into nothingness; they were trained and supple. Her skin was bronzed from the tanning bed and covered with tiny sweat beads that shimmered like mother-of-pearl. The long, black curly hair had randomly arranged itself over the white paper pad as if a photographer were shooting pictures for Vogue magazine. Her face was unmistakably Arabic, with distinctive features, deep black eyes below dark eyebrows that revealed her unveiled sensuality to any man. Her full breasts were flawless, accentuated by her slim waist, underneath expressive hips.\nAn ethereal, dissatisfied smile played on her face.She hadn’t expected this. In this state of confusion, she didn’t know if she was awake or dreaming. She had wandered between worlds too many times. What should she do now? She raised her hand to her face and stretched out three fingers.\nThere was the tool of her revenge, in her hand - a syringe. Everything in her head became one, the world around her narrowed down to this tiny needle.\nUncertain about what was supposed to happen, she listened more closely to herself. The sequence of a dream, as she knew it.

This is the response:

what it was. A nightmare. She couldn’t do this. She couldn’t kill him. The thought of murder was overwhelming, and it seemed that with each passing moment, it became more and more impossible.
With a deep breath, she threw the syringe across the room, and it clattered against the tiled floor. Her heart was racing, but she knew she had made the right decision.
She slowly sat up and picked up her clothes from the floor. She dressed herself in silence, still feeling the weight of her disappointment. She knew she couldn’t stay here any longer.
As she walked towards the door, she took one last look around the room, memorizing every detail. She didn’t know why, but she had a feeling that she’d never be back here again.
As she stepped out into the hallway, she felt the cool air hit her skin, and her senses sharpened. She was free. She was alive.
Rajna walked towards the exit, the sound of her footsteps echoing through the halls. She didn’t know where she was headed, but she knew that wherever it was, she was ready to face it.sense of control. But soon enough, Rana realized that her behavior was not acceptable, and she needed to change. She started going to the gym and significantly reduced her junk food intake. As a result, she lost weight, and her self-esteem improved.

I have to repeat: The application was running nice before. Here I used temperature 0.7 and 1.0. No penalty, model ChatGpt3_5Turbo

Edit and proofread… Those are two separate tasks. Edit is not defined at all. Proofread might need a better definition.

I would separate those in two distinct API calls after being more precise on task.

I notice that when working with long chunks of text, depending on the request format and task the model can lose “focus” on the task and get “trapped” in the content of the text to edit. As a result you get unpredictable output.

Try following:

Proofread:

System Prompt Example 1:
You are an assistant specialized in proofreading and editing. Your task is to correct language mistakes, adjust sentence lengths for better understanding, replace repeated language structures, and adapt the text to match the provided style. Don't comment on the user's input, just make the necessary edits and return the result.

User Prompt Example 1:
Please proofread and edit the following text to match a formal academic style: "Hey there! So, the thing is, you know, climate change is kinda like a really big deal. People gotta do something about it, like, ASAP."

System Prompt Example 2:
You are an assistant who excels in proofreading and language editing. Your role is to rectify any language and syntax errors, modify the sentence length to ensure simpler thoughts workflow, eradicate any language structure repetitions, and mold the text according to the specified style. Remember not to comment on the user's input, but just to amend and return the output.

User Prompt Example 2:
Could you proofread and edit this piece of text to suit a casual blog style: "It is of utmost importance to understand the implications of rapid deforestation. Immediate countermeasures must be taken to prevent the catastrophic consequences of the same."

System Prompt Example 3:
You are an assistant with expertise in proofreading and language adjustment. Your task is to fix any language and syntax mistakes, alter sentence lengths to enhance understanding, substitute repeating language structures, and adjust the text to align with the provided style. Avoid providing any comments on the user's input, only return the revised result.

User Prompt Example 3:
Please edit and proofread the following text to fit a children's storybook style: "In our complex biosphere, the extinction of species due to human activities is accelerating at an alarming rate, threatening biodiversity."

For the user prompt, the format should clearly specify the text to be edited and the desired style. Something like: “Please edit the following text to match the [desired style]: [text to be edited]”. The more specific the user is with their request, the better the assistant can adapt the text to the desired style.

As the bot says: mind wrapping the text to edit in an appropriate user message template. And avoid long text… Cut it into sections separated by titles

Thank you for taking the time to participate in this discussion. I’m in agreement that simple and clear prompts yield better results with large language models (LLMs). However, as it relates to the current issue we’re discussing, it’s important to remember that the application was operating efficiently not too long ago.

Despite experimenting with numerous prompts and adjusting various parameters to optimize text output, the results remain unchanged. Even a simple “Proofread” prompt functions well until the token count is increased once again. This issue persists not just in the API but also in the playground.

While the practice of ‘humanizing’ prompts may make our interaction with the LLM more natural, it’s important to recognize that these models don’t require such language nuances. For instance, using polite phrases like “Please” or “Could you…” is essentially superfluous. If we consider a scenario where the same prompt is invoked 500 times, just think of the number of tokens being consumed due to this unnecessary verbosity.

Lastly, it’s crucial to note that while the prompts may be engaging, they aren’t addressing the root of the issue. Our main problem still persists. Let’s refocus our efforts on identifying and resolving this core issue.

Here is more information with three tests. I was below maxToken = 4096/2 with TotalTokens = 825. Top P and Temperature were 1. Both penalty 0. Maximum length 2048. In Playground the result is correct. As you can see the result text was shortened and the text seems to be halluzinated. When I stream the completion it is the same result.

Prompt: You are a English literature professor. Proofread the text.

Content: “Rajna saw herself from the outside, and it was indeed her own body because she could feel her disgust, touch herself, stare at herself incredulously. Naked, she sat on a cot in the white room where she had to taste the pain. It smelled like disinfectant.\nIn the brown corduroy pants over the chair, a pager had beeped. Shortly afterward, the parquet floor trembled under the weight of his quick steps. Without a word, he had left and closed the door behind him. The key turned energetically in the lock.\nAs soon as he was gone, the young woman fell back annoyed onto the examination table and stared desperately up at the ceiling.\nShe gagged. Saliva filled her mouth.\nHer skin was reddened and sweaty, as if she had been lying in a sauna after an infusion. She was so close to her goal, but now…she should have massacred him right away.\nHer body had become the battlefield of her senses, not searching for love, but diligently collecting the nectar in her brain greedily, from which our bodies boil with angry obsession. When memories of past thoughts touched her lips, when her vagina was filled, when sweaty bodies slid over each other, when the memories whipped her up, then the desire for his death surged in her mind. It was comparable to the effect of ecstasy; that intense euphoria, that high feeling that turns virgins into warriors, dispels any fatigue, forgives any hunger.\nRajna lay naked on the narrow bed with her arms and legs outstretched. Her limbs hung down into nothingness; they were trained and supple. Her skin was bronzed from the tanning bed and covered with tiny sweat beads that shimmered like mother-of-pearl. The long, black curly hair had randomly arranged itself over the white paper pad as if a photographer were shooting pictures for Vogue magazine. Her face was unmistakably Arabic, with distinctive features, deep black eyes below dark eyebrows that revealed her unveiled sensuality to any man. Her full breasts were flawless, accentuated by her slim waist, underneath expressive hips.\nAn ethereal, dissatisfied smile played on her face.She hadn’t expected this. In this state of confusion, she didn’t know if she was awake or dreaming. She had wandered between worlds too many times. What should she do now? She raised her hand to her face and stretched out three fingers.\nThere was the tool of her revenge, in her hand - a syringe. Everything in her head became one, the world around her narrowed down to this tiny needle.\nUncertain about what was supposed to happen, she listened more closely to herself. The sequence of a dream, as she knew it. Yes, that’s it. This state, that ‘being in my head’, was literally felt again.\nAm I dreaming again? She was never sure of that, but today it couldn’t be.\nWhenever she didn’t know if she was dreaming or awake, she compared objects from her surroundings with her”

Result:

{ChatCompletionCreateResponse { ObjectTypeName = chat.completion, Successful = True, Error = , Model = gpt-3.5-turbo-0301, Choices = System.Collections.Generic.List`1[OpenAI.GPT3.ObjectModels.SharedModels.ChatChoiceResponse], Usage = UsageResponse { PromptTokens = 632, CompletionTokens = 193, TotalTokens = 825 }, CreatedAt = 1686716559, Id = chatcmpl-7RCJjeuEagfjC4UdNiNBCTp09qDhp }}

{ChatCompletionCreateResponse { ObjectTypeName = chat.completion, Successful = True, Error = , Model = gpt-3.5-turbo-0301, Choices = System.Collections.Generic.List`1[OpenAI.GPT3.ObjectModels.SharedModels.ChatChoiceResponse], Usage = UsageResponse { PromptTokens = 632, CompletionTokens = 261, TotalTokens = 893 }, CreatedAt = 1686716926, Id = chatcmpl-7RCPeNXR8ylfpUqS93eaBROkhF5kg }}

{ChatCompletionCreateResponse { ObjectTypeName = chat.completion, Successful = True, Error = , Model = gpt-3.5-turbo-0301, Choices = System.Collections.Generic.List`1[OpenAI.GPT3.ObjectModels.SharedModels.ChatChoiceResponse], Usage = UsageResponse { PromptTokens = 641, CompletionTokens = 249, TotalTokens = 890 }, CreatedAt = 1686718089, Id = chatcmpl-7RCiPsxY4j045KOw1Sw9pd3fic4Lx }}

Result Text: “memories and knowledge. The white room was familiar, and the painful memories were still fresh. She slowly realized that she wasn’t dreaming, and the syringe in her hand was real.\nHer thoughts raced, and a plan formed. She knew what she had to do to get her revenge. She injected the contents of the syringe into her arm and waited. The pain she had been enduring disappeared, replaced by a euphoric feeling.\nShe gathered her clothes and left the room with purpose. She knew what she needed to do next, and nothing could stop her. She had found the strength to take action, to take control of her life and her fate.\nAs she walked out into the world, ready to exact revenge on the one who had hurt her, she felt powerful, unstoppable. She had found a way to turn her pain into fuel, to transform herself from a victim into a warrior. And she knew that nothing could stand in her way.”

Result Text: “memories. In this case, she looked at the syringe in her hand, and it felt solid and real. She knew what she had to do.\n\nRajna took a deep breath and injected the contents of the syringe into her arm. Within seconds, she felt a rush of energy and power that made her feel invincible. She stood up from the bed, her body pulsing with an electrifying force.\n\nWith a newfound determination, she walked out of the white room and into the hallway. Her steps were measured and purposeful, as if she had a mission to accomplish. She passed by several closed doors, hearing muted voices and beeping machines behind them. But she did not stop to look inside.\n\nFinally, she reached the end of the hallway, where a large metal door stood in her way. She pushed it open with ease, the metal groaning under the pressure.\n\nBeyond the door, a blinding light filled her vision. Rajna stepped forward, her heart pounding in her chest. As she emerged into the sunlight, she saw a vast desert landscape stretching out before her.\n\nShe knew what she had to do. She had to find him and avenge herself.\n\nRajna took a deep breath and began to walk forward, into the unforgiving desert.”

Result Text: “own memories. Her eyes searched the room, trying to find something that could prove to her that she was truly awake. She noticed a crack in the ceiling, a small spider crawling up the wall, and a piece of lint on the floor. These real and mundane objects brought her back to reality.\n\nRajna took a deep breath and tried to calm herself. She couldn’t afford to be lost in her thoughts anymore, not when vengeance was so close. She filled the syringe with a clear liquid and stared at it for a moment, her mind racing with possibilities.\n\nShe would make him pay for what he had done to her. She would show him that she was not weak, that she was in control. With a steady hand, she injected the contents of the syringe into her thigh.\n\nAs the liquid entered her bloodstream, she felt a rush of adrenaline and power. She was in control now, and nothing would stop her. She got off the bed and put on her clothes, feeling confident and ready for anything.\n\nShe walked out of the room, ready to face whatever challenges lay ahead. And as she stepped out into the bright light of day, she knew that she had finally broken free from the chains of her past.”

1 Like