Benchmarking in a vacuum isn’t interesting to any real product. I am measuring end user experience.
User presses button. User has to wait until they see the result. How long does this take?
In your situation with an end user waiting, I would stream the answer back to them. The first token latency is a few seconds, just like it is in the Playground.
Above we were mostly taking last token latency and/or token output rates per second, and not first token streaming latency, which is probably the most interesting performance parameter for your application.
I already said that I must post process the data before I can show it.
(One common case of this is moderation solutions.)
So, no, I can in fact not stream the data, as I already said, and token generation rate directly impacts the latency to result for the user, as I already said, so generation at significantly faster than reading speed is both useful and important, as I already said.
Another case where generation faster than human reading speed is also important, is when generating steps to execute, such as in plugins.
So, I stand by the assessment that GPT-4 generation speed currently is a significant impediment.
I cannot find anything in OpenAI docs to support this. They only mention moderating to " You can use our free moderation endpoint and safety best practices to help you keep your app safe."
So I could interpret this to mean, in your case, stream to your users, then send the full result to moderation (after it’s done), and if it comes back flagged, the put the user on cooldown, or something similar, and see what happened.
I think you are taking something too literally, but feel free to share where you have to moderate everything before it goes out. If this were the case, the streaming feature is in violation of that policy. This is a contradiction.
But if you are referring to some other moderation or post processing thing, then yeah, as a batch, you aren’t getting past last token latency. But no AI model will have great responsive last token latency, at least for the next 6 months, until sub-quadratic models, that work and are good (and running on smoking hot FPGA’s or ASIC’s), come out.
I understand the argument, end user latency is a combination of multiple things, I’m sure people are aware of this. Benchmarking one thing separately still provides us with relevant information, that can be used by other people.
@jwatte I’m sorry if my information about human reading speed it’s relevant to you, it may be for other people, if you’re having issues with latency I’ll advise that you create a separate topic about this, as it will aid other people in helping you. This thread is for the (very-small) early rollout of GPT-4-32K
I’m expecting most people will be using the 32k while streaming to the end user, running an agent on GPT-4-32k is just too expensive 
1 Like
What a weird thing to say. It’s absolutely critical to benchmark all components of a process in order to understand where bottlenecks are occurring.
2 Likes
My latest use of 32k is to have long conversations with GPT and to analyze legal terms of service agreements. So more of a onesie-twosie thing, not production.
But I have been running 32k in parallel with 8k for model redundancy, randomly for 25% of all traffic. Nothing earth shattering or mission critical, just redundancy to get data out without retries.
4 Likes
Ha ha, what a coincidence, I’ve been using it for legal texts as well, more specifically to add legal explanations to specific context, very useful stuff
1 Like
This is indeed the case – I was not referring to “the OpenAI moderation API;” I was referring to certain moderation situation that need to see the full context. (As mentioned before, …)
Sure! I’m answering the argument that said that I was wrong when I said that, in the use cases I have, generation speed matters, and generation above human-reading-speed is important.
In runtime, currently, yes absolutely it’s too expensive!
In dollars, GPT-4 is cheap compared to what I would be willing to pay if I could get fine tuning and guaranteed performance.
1 Like
Only the paid api (which I have)? Not the chat window? Are you getting full 32K in the chat window?
I don’t have ChatGPT, so no idea. The 32k is, for me, through the API. And I can use 32k in the Playground for quick testing purposes.
There’s a much shorter limit in the chat window.
I have paid ChatGPT, but even so, the limit to the amount of text I can paste in a chat window is probably around 2000 tokens?
When I have something longer, I have to kick up the Python terminal and paste a request in there instead. Somewhat annoying but not the end of the world.
I am more concerned about that number. Some one forgot to add float parsing/ rounding?
1 Like
Who doesn’t like sub-picosecond timestamps? 
Here, “Time taken: 38.4 seconds”.
Happy now?
2 Likes
It just shows high ambitions for the future 
2 Likes
The power of large language models is immense. Crazy to think where this will be in a year.
2 Likes
Absolutely can’t wait! Every day now something new and amazing to discover. 
1 Like
WOW GPT4 32K would be a game-changer for me , how can I convince them to open the gate for me?
I’m really looking forward to it, but my application for the GPT-4 API has still not been approved. Could you help me get it through (org-62SU5I2LI2FRYTUhW2N5oCQN)? Thanks a lot!
1 Like
I have paid ChatGPT, but even so, the limit to the amount of text I can paste in a chat window is probably around 2000 tokens?
A workaround could be to just tell it you’ll paste it in multiple parts. Though that would probably involve manually splitting it into short enough chunks which would be extra work.
Not sure if anyone would find this helpful but I’ve been working on a basic python script for interacting with the API that has some features like automatic chat logging, conversation saving/loading, text box pop up for pasting lots of text, etc
3 Likes