Why is is that Open AI advertises 128k context windo for GPT-4 turbo and GPT-4o models when the actual token output is a very limited. My prompt lenght and input rarely exceed 2500tokens yet the output is short (i.e. 800 tokens.) and many times truncated. The more I increase the input, the less the output lenght is. What am I doing wrong? Feels like Open AI is throtlting back output.
3 Likes
Welcome to the Community!
The context window refers to the sum of input and output tokens that a model can handle / produce. Both the gpt-4 (incl. gpt-4o) and the gpt-3.5 model series are limited to 4096 output tokens. That leaves technically close 124,000 for input tokens.
In practice you are right that we are frequently seeing outputs well below the 4096 tokens. That said, how many output tokens are produced depends entirely on your prompt. There is no correlation between the number of input tokens and the resulting output tokens. It all comes to down to how you word the prompt.
Feel free to share an example prompt you are using and we can try to identify possible reasons that limit your output length.
5 Likes
I dont think that putting the new model available for free (on account of competing with meta AI) is a great idea. First a free user with several accounts will have more than a paying user (thats kinda shooting in the foot) , second , free users fill the model with garbage information that will affect the model (again, supporting the economic failure of the business) ; third , servers outage will be a growing thing (they already are a problem) ; fourth, how will Open AI compesate for paying users leaving ? (Chat gpt will be the next Gemini?) ; fifth , does Open AI think that people paying use the Chat to find out cooking recipes or the nearest restaurant? ; last, why will anyone pay for something that is free ?
Quality will drop like a rock, along with a lot of money . Good thing that Claude just got available in Europe today. I just hope that Open AI makes a big turn on this mistake
3 Likes
Getting access to the audio capabilities in the API will be a massive boost, currently use whisper to transcribe 3.5 turbo to classify and then 4 turbo to clean and correct whispers transcription errors and analyse the text. Being able to do this in one step with a multimodal model will be a game changer
@openai I will definitely put my hand up to beta test the audio endpoint
3 Likes
Thank you for the interesting updates and best of luck with your future developments.
I am curious if the audio analysis will support continuous data streams. This would mean that developers (e.g., in Python) would no longer have to manage wake-up words, recording, silence detection, user speech detection, creating local audio files, and sending them to Whisper. This capability could enable developers to create applications as fascinating as those seen in demos.
Thanks a lot.
1 Like
128 K context window for half the price and improved rate limits, take my money 
. Still need to test it with Assistants API for batch image processing to check the performance if it matches up with expectations.
Subject: Urgent: Unable to Access New GPT-4O Features Despite Being a Long-Term GPT Plus Subscriber
Dear OpenAI Support Team,
I hope this message finds you well. I am experiencing significant issues accessing the new features introduced in GPT-4O, as announced on May 13, 2024. Despite being a GPT Plus subscriber for over a year, I am unable to enable the audio and video options in my settings. Currently, my settings only show beta features for browsing, DALL-E, and code.
Here are the steps I have taken:
- Verified that my Microsoft Edge browser is up-to-date.
- Restarted both my computer and the ChatGPT session.
- Checked and ensured my subscription is active and that I am logged into the correct account.
Despite these efforts, I cannot access the new features. I am keen to experience the improved, low-latency interactions and the ability to interrupt conversations without losing coherence, as demonstrated by OpenAI.
Will I be provided with any help, technical support, or any type of solution?
Thank you for your time and assistance.
Best regards, George
1 Like
It doesn’t appear so. I always verify in the Playground since this crucial information often seems missing.
We recognize that GPT-4o’s audio modalities present a variety of novel risks. Today we are publicly releasing text and image inputs and text outputs. Over the upcoming weeks and months, we’ll be working on the technical infrastructure, usability via post-training, and safety necessary to release the other modalities. For example, at launch, audio outputs will be limited to a selection of preset voices and will abide by our existing safety policies. We will share further details addressing the full range of GPT-4o’s modalities in the forthcoming system card.
1 Like
What is the best way to request early access to the “Conversational Mode” I have a friend who is in recovery after a critical surgery in the hospital. I would love to help him enable this feature to help him with his recovery.
You cannot. There is no way to get early access to any of these features.
These features are currently being Red-Teamed. In theory, you could gain access to them by applying for and being selected to join the network of Red-Team users, but even then you would be under an NDA and be unable to share any information with your friend and you absolutely wouldn’t be allowed to give them access to the new features.
2 Likes
Thanks for the rapid response I guess I’ll have to be a little patient and wait for it to be rolled out to us, exciting news though!
1 Like
For some reason I am not getting the 4o capabilities and it is still operating exactly like 4 despite saying there’s a new model. None of the features announced yesterday are available to me–even when I toggle to 4o
1 Like
I am paying but still not getting access to the 4o capabilities.
1 Like
@andremr I didn’t actually know that turbo had a 128k limit… nor did I know anything about more efficient embeddings and tokenization, leave it to @_j to understand and explain that stuff instantaneously.
@kevgcampbell I’m using teams and I think we’re all seeing rolling updates which are probably made a bit slow by everyone trying it out at the same time.
My ChatGPT interfaces has changed today, and has anyone else seen that they’re starting to roll out the connections to Google Workspace and Microsoft? I had a pop-up in my UI, but it doesn’t work yet. lol
This is great. ^^
Team, exciting times but just a suggestion from a long time fanboy. When the star of your announcement show is the chat interface, perhaps state that it’s coming but not quite here yet. Maybe even give an ETA. I get it that Google was on your heels (no need to confirm that, we all know) and that’s understandably promoted the early announcement, but just tell us to save us running around reading every thread and blog to find the answer. Gpt4o is exciting and we all wish you the best for its success. xo
1 Like
They did say it several times during the 30 minute event and it’s stated again at least a few times on the announcement page.
I’m not sure what more they could do.
We recognize that GPT-4o’s audio modalities present a variety of novel risks. Today we are publicly releasing text and image inputs and text outputs. Over the upcoming weeks and months, we’ll be working on the technical infrastructure, usability via post-training, and safety necessary to release the other modalities. For example, at launch, audio outputs will be limited to a selection of preset voices and will abide by our existing safety policies. We will share further details addressing the full range of GPT-4o’s modalities in the forthcoming system card.
I appreciate your guidance… its helpful. So with GPT4 Turbo, my prompt is 750 tokens; input is 2000 tokens, but Im getting only around 1000 tokens of output. Shoudn’t I have around 3346 tokens avialable for output? (I am getting only 1/3 of that even when we have the limit set to the max of 4096 tokens.) The math isn’t math-ing for me. 2k tokens of outputs for this prompt. Thanks again!
In your case, you would have the full 4096 tokens available for output. Prompt plus context that you supply are both considered input tokens.
As said, it really depends on how what task you are asking the model to complete and how you phrase your prompt. Sometime a single word or sentence can make a significant impact on the number of output tokens.
I re-emphasize this again: just because you supply a large number of input tokens does not automatically mean you’ll get a large number of output tokens.
If you can share your actual prompt, we can see what might be causing it. But without that information, it is difficult to provide pointers on how might be able to increase your output.
1 Like
Yes, you can access it if you’re selected as one of their ‘internal’ users. So yeah, 
Unfortunately, the NDA is blocking the information from spreading or facing the legal consequences. So yeah, good luck. The newest GPT-4o is awesome though not everyone can taste its full capabilities. For normal folks and even power users, they are “throttled”. You need to be their insiders to access.