up until 48 hours ago, when submitting fine tuning jobs, they completed fairly quickly.
now when we submit them, they never finish.
openai api fine_tunes.follow -i ft-xxx
[2022-06-24 07:41:59] Created fine-tune: ft-xxx
[2022-06-24 07:43:28] Fine-tune costs $0.60
[2022-06-24 07:43:28] Fine-tune enqueued. Queue number: 31
this job a few days ago would have completed in 20/30 minutes.
a job we submitted yesterday never completed so we cancelled it.
what’s up? is this normal?
As a follow up from yesterday’s outage, the fine tuning jobs got delayed. We’re working through the queue now
yup. almost done:
openai api fine_tunes.follow -i ft-xxx
[2022-06-24 07:41:59] Created fine-tune: ft-xxx
[2022-06-24 07:43:28] Fine-tune costs $0.60
[2022-06-24 07:43:28] Fine-tune enqueued. Queue number: 31
[2022-06-24 10:32:46] Fine-tune is in the queue. Queue number: 28
[2022-06-24 10:32:46] Fine-tune is in the queue. Queue number: 28
[2022-06-24 10:32:46] Fine-tune is in the queue. Queue number: 28
[2022-06-24 10:32:53] Fine-tune is in the queue. Queue number: 27
[2022-06-24 10:32:58] Fine-tune is in the queue. Queue number: 26
[2022-06-24 10:47:02] Fine-tune is in the queue. Queue number: 25
[2022-06-24 10:47:08] Fine-tune is in the queue. Queue number: 24
[2022-06-24 10:48:20] Fine-tune is in the queue. Queue number: 23
[2022-06-24 10:49:09] Fine-tune is in the queue. Queue number: 22
[2022-06-24 10:50:29] Fine-tune is in the queue. Queue number: 21
[2022-06-24 11:03:58] Fine-tune is in the queue. Queue number: 20
[2022-06-24 11:10:56] Fine-tune is in the queue. Queue number: 19
[2022-06-24 11:18:46] Fine-tune is in the queue. Queue number: 18
[2022-06-24 11:18:51] Fine-tune is in the queue. Queue number: 17
[2022-06-24 11:20:11] Fine-tune is in the queue. Queue number: 16
[2022-06-24 11:23:01] Fine-tune is in the queue. Queue number: 15
[2022-06-24 11:23:09] Fine-tune is in the queue. Queue number: 14
[2022-06-24 11:23:13] Fine-tune is in the queue. Queue number: 13
[2022-06-24 11:26:01] Fine-tune is in the queue. Queue number: 12
[2022-06-24 11:27:02] Fine-tune is in the queue. Queue number: 11
[2022-06-24 11:27:06] Fine-tune is in the queue. Queue number: 10
[2022-06-24 11:27:11] Fine-tune is in the queue. Queue number: 9
[2022-06-24 11:38:34] Fine-tune is in the queue. Queue number: 8
[2022-06-24 11:45:32] Fine-tune is in the queue. Queue number: 7
i will check it again in another four hours.
thx.