Considering the default timeframe is 24 hours, a push connection open for you for half the day may not be that practical. An expectation that you are immediately queued for prompt execution might be disappointment.
Or, it is easy to speculate, move anything they can off of peak times, and reward those who will wait for idle time with the lower price. You can look at the daily and weekly response times and see exactly when these should run, no algorithm needed. Then add an empty queue of fine-tuning and other servers that might switch to language model.
Also, that off-peak is likely the reason for a 24 hour window; the best “window” is really when the world sleeps.
Good for that benchmark that you can pick up from your inbox tomorrow.
There’s a few things that aren’t clear, and we hope the big brains have considered:
- Tiered rate limits: everything will be managed, never going over and/or never affecting production rates? edit: documented to be completely separate, only limit is queue in tokens.
- Moderations: You wouldn’t want 1000 system prompts that are at a triggering level all going through to accumulate against you (aka check inputs first), or paying for 1000 inputs that get you no more than
content_filterstop reason… - Account balances. When you are emptied halfway through or hard limit is hit (if that even works any more…several have reported huge overages), what is expected? Termination of the batch, or it still keeps running on the remainder…(it seems the latter is likely, you just get a file of errors for what couldn’t be paid at the time they ran)