It looks like GPT-4-32k is rolling out

Benchmarking in a vacuum isn’t interesting to any real product. I am measuring end user experience.
User presses button. User has to wait until they see the result. How long does this take?

In your situation with an end user waiting, I would stream the answer back to them. The first token latency is a few seconds, just like it is in the Playground.

Above we were mostly taking last token latency and/or token output rates per second, and not first token streaming latency, which is probably the most interesting performance parameter for your application.

I already said that I must post process the data before I can show it.

(One common case of this is moderation solutions.)

So, no, I can in fact not stream the data, as I already said, and token generation rate directly impacts the latency to result for the user, as I already said, so generation at significantly faster than reading speed is both useful and important, as I already said.

Another case where generation faster than human reading speed is also important, is when generating steps to execute, such as in plugins.

So, I stand by the assessment that GPT-4 generation speed currently is a significant impediment.

I cannot find anything in OpenAI docs to support this. They only mention moderating to " You can use our free moderation endpoint and safety best practices to help you keep your app safe."

So I could interpret this to mean, in your case, stream to your users, then send the full result to moderation (after it’s done), and if it comes back flagged, the put the user on cooldown, or something similar, and see what happened.

I think you are taking something too literally, but feel free to share where you have to moderate everything before it goes out. If this were the case, the streaming feature is in violation of that policy. This is a contradiction.

But if you are referring to some other moderation or post processing thing, then yeah, as a batch, you aren’t getting past last token latency. But no AI model will have great responsive last token latency, at least for the next 6 months, until sub-quadratic models, that work and are good (and running on smoking hot FPGA’s or ASIC’s), come out.

I understand the argument, end user latency is a combination of multiple things, I’m sure people are aware of this. Benchmarking one thing separately still provides us with relevant information, that can be used by other people.

@jwatte I’m sorry if my information about human reading speed it’s relevant to you, it may be for other people, if you’re having issues with latency I’ll advise that you create a separate topic about this, as it will aid other people in helping you. This thread is for the (very-small) early rollout of GPT-4-32K

I’m expecting most people will be using the 32k while streaming to the end user, running an agent on GPT-4-32k is just too expensive :laughing:

1 Like

What a weird thing to say. It’s absolutely critical to benchmark all components of a process in order to understand where bottlenecks are occurring.


My latest use of 32k is to have long conversations with GPT and to analyze legal terms of service agreements. So more of a onesie-twosie thing, not production.

But I have been running 32k in parallel with 8k for model redundancy, randomly for 25% of all traffic. Nothing earth shattering or mission critical, just redundancy to get data out without retries.


Ha ha, what a coincidence, I’ve been using it for legal texts as well, more specifically to add legal explanations to specific context, very useful stuff :laughing:

1 Like

This is indeed the case – I was not referring to “the OpenAI moderation API;” I was referring to certain moderation situation that need to see the full context. (As mentioned before, …)

Sure! I’m answering the argument that said that I was wrong when I said that, in the use cases I have, generation speed matters, and generation above human-reading-speed is important.

In runtime, currently, yes absolutely it’s too expensive!

In dollars, GPT-4 is cheap compared to what I would be willing to pay if I could get fine tuning and guaranteed performance.

1 Like