How does multi-turn conversation work in the o1-preview Chat Completion API?

API
3

Welcome @jarrelscy1

Per the docs, reasoning tokens are discarded:

How reasoning works

The o1 models introduce reasoning tokens. The models use these reasoning tokens to “think”, breaking down their understanding of the prompt and considering multiple approaches to generating a response. After generating reasoning tokens, the model produces an answer as visible completion tokens, and discards the reasoning tokens from its context.

Here is an example of a multi-step conversation between a user and an assistant. Input and output tokens from each step are carried over, while reasoning tokens are discarded.

The image visually explains how the input, reasoning, and output processes work across multiple turns within a set context window of 128k tokens, highlighting that the reasoning is discarded with only input and output retained from the previous turn. (Captioned by AI)
The image visually explains how the input, reasoning, and output processes work across multiple turns within a set context window of 128k tokens, highlighting that the reasoning is discarded with only input and output retained from the previous turn. (Captioned by AI)1700×1234 54.1 KB

2 Likes