I have been testing but it is really confusing.
I will give some examples.
For reference, I am calculating my tokens like this. This is coming directly from OpenAI’s Cookbook.
def num_tokens_from_messages(messages, model="gpt-4o-mini-2024-07-18"):
"""Return the number of tokens used by a list of messages."""
try:
encoding = tiktoken.encoding_for_model(model)
except KeyError:
print("Warning: model not found. Using o200k_base encoding.")
encoding = tiktoken.get_encoding("o200k_base")
if model in {
"gpt-3.5-turbo-0125",
"gpt-4-0314",
"gpt-4-32k-0314",
"gpt-4-0613",
"gpt-4-32k-0613",
"gpt-4o-mini-2024-07-18",
"gpt-4o-2024-08-06"
}:
tokens_per_message = 3
tokens_per_name = 1
elif "gpt-3.5-turbo" in model:
print("Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0125.")
return num_tokens_from_messages(messages, model="gpt-3.5-turbo-0125")
elif "gpt-4o-mini" in model:
print("Warning: gpt-4o-mini may update over time. Returning num tokens assuming gpt-4o-mini-2024-07-18.")
return num_tokens_from_messages(messages, model="gpt-4o-mini-2024-07-18")
elif "gpt-4o" in model:
print("Warning: gpt-4o and gpt-4o-mini may update over time. Returning num tokens assuming gpt-4o-2024-08-06.")
return num_tokens_from_messages(messages, model="gpt-4o-2024-08-06")
elif "gpt-4" in model:
print("Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.")
return num_tokens_from_messages(messages, model="gpt-4-0613")
else:
raise NotImplementedError(
f"""num_tokens_from_messages() is not implemented for model {model}."""
)
num_tokens = 0
for message in messages:
num_tokens += tokens_per_message
for key, value in message.items():
num_tokens += len(encoding.encode(value))
if key == "name":
num_tokens += tokens_per_name
num_tokens += 3 # every reply is primed with <|start|>assistant<|message|>
return num_tokens
BATCH JOB COMPLETED
Here I can compare my calculation of tokens that I am sending. This is because in the output file generated after a batch job is completed, you can retrieve the prompt tokens that OpenAI calculated.
- Calculated tokens by myself: 70,058
- Calculated tokens by OpenAI in the output: 87,055
New single batch job
I have created a new file for a batch job with 149,669 tokens. For this time, I have included the max_completion_tokens in the body of each task written in the file. Also, to my calculation of tokens, I have added this 128 since they should also count for the enqueued tokens. This batch job went also from ‘validating’ to ‘failed’ without me having any other running batch job.
I have checked my files for right encoding, and they are correctly encoded with UTF-8. There are no errors. Also, none of my previous batch jobs are still in a status that could indicate ‘enqueued’. They are either completed or failed.
Sadly, I have no ideas left. Also, no response from support.
UPDATE
It really seems like an issue to me because I have submitted another batch job with only 16,329 tokens and it goes directly to ‘failed’ even without having any running jobs at all.
I have parsed the .jsonl file that I used for this batch job, and everything is correctly encoded and formatted. I sent each of the defined task to the chat completions using the openai python library and every single request worked.
I will try to contact any other kind of support from OpenAI.
