The Batch API has a limit on the number of tokens that can be enqueued at any one time. If you try to add another batch to the queue, and it would exceed your limit, then your batch fails.
The only solution to this is to write your code so that it submits one batch of the maximum size, waits for it to finish, and then submits the next batch. This doesn’t make any sense. Why does the batch system not have a queue?
It’s not as though storage is the limiting factor, because you can already upload all the batch files up front, without hitting an arbitrary limit.
Please add a queue to the batch processing system!