Hello there,
Am just wondering about how the token system really works? Does it count tokens based on inputs or outputs.
In anyway, i have large sets of data in text formats, 1 txt file is about 40.000 tokens, 100 txt files = 4.000.000 tokens.
Is it possible to use tire 1 gpt 4 for that, extracting structered data from unstructered text worth of 4 million tokens?
Hello there to you!
The rate limiter considers both input and output tokens.
However, your concern is more about the model capabilities.
The AI is limited in how much data it can ultimately contemplate and analyze at once. The AI model has a context window length, the maximum memory of tokens, an area for both processing your language input and for forming a response.
Standard high-quality gpt-4 has a context length of 8k tokens. gpt-4-turbo has a context length of 125k for understanding, with a limited output.
You need to partition your job into smaller tasks, or load only data that is relevant.
As for rate limits: At tier 1 (paying less than $50 in the past),
gpt-4-turbo-preview has a limit of 150000 tokens per minute. However it has a much more restrictive 500000 tokens per day.Both limits are sufficient for sending a maximum context length request via chat completions once per minute.
Now that makes more sense, much clearer and appreciated, thank you