The parameter n specifies how many chat completion choices to generate for each input message. How does it work? Does the prompt get sent n times? For long prompts, it may hinder the token rate limit. Or, does it do it in some optimized way (i.e. send the prompt once, and generate n completions)? …

You only “send” once, the only change to the API call is the value of the N number, and you are only charged for the extra tokens of the extra output cases. While technical details aren’t made available, the most logical way is to load the input into the context of the model, and simply capture, wi…

How does `n` parameter work in chat completions?

API

theofficialjeffan December 9, 2023, 7:44am 6

wait so just to confirm, with a low n, the input tokens are only counted once in terms of the input token limit and not n * tokenizer(input prompt string)?

Topic		Replies	Views
Questions on setting n and max_token API	4	904	March 20, 2024
Clarification on token pricing for multiple completions (n>1) in a single API call" API pricing	1	414	July 3, 2024
Do I need to increase `max_tokens` when using `n>1` e.g. `n=3` for generating multiple chat completions API	8	2063	July 2, 2023
Is the max_tokens parameter of the completions endpoint applicable for ALL or EACH response? API	7	2270	July 3, 2023
Multiple prompt responses everywhere API	6	3640	December 25, 2023

How does `n` parameter work in chat completions?

Related topics