Does response/generation time of gpt 4 depends on size of input prompt?

Community
1

Hi, this is in relation to the response/generation time for gpt-4. I am mostly concerned with the performance related to api calls. This has 2 parts.

Q1 :
While streaming, does the time to get the response of first generated token is related to the size of input prompt?

Q2 :
Lets say the model generates 100 tokens for 2k input tokens, will it take more time to generate the same 100 tokens for 10k input tokens?

I know that should typically be the case, but i want to know others views as well on this.

3

my bad, I have updated the question for you

4

There is a delay related to the size of the input token size, although this is typically only a fraction of the total response time. I do not have access to the 32k model to build any sort of data based model of the time taken, but from the 4k and 8k models of 3.5 and 4 the increase seems small, on the order of a second or so for max token contexts.