Python test code to run (and slowness measure confirmed by another)
I’m still doing well, compare the “latency” of 1 token to a full response of 512:
Title
[1 tokens in 1.0s. 1.0 tps]
Title: Embracing Digital Transformation: Unlocking the Power of the Digital Age
[128 tokens in 1.9s. 67.6 tps]
Title: Embracing Digital Transformation: Unlocking the Power of the Digital Age
[512 tokens in 7.2s. 70.8 tps]
post-pay, Western US.
Unlike other reports of massive slowing in the last few days:
So this is not a “blame on intermittent stuff and the user”.
Although it does appear to be “sticky” to particular users. Reports of where you are geographically connected, whether you are prepay or billing or free trial, whether you ever paid a bill, etc. could help determine why some are affected and some are fast.