How to set temperature and other sampling parameters of model in Open AI Assistant api?

Our use case, of creating deterministic text also requires the ability to adjust temperature. For now, we will move away from Assistants and over to the chat completion API.

hello sir,
I am currently using assistant api with retrieval tools there I am providing file with user message. I have a query regarding token count as in response from api token count was around 20k-30k and user message has around 200 token only so what is the internal mechanism so that they are getting increased.

I love the Assistant Api; thanks for this great feature.

We very much also need to make the answers more deterministic. We are providing suggestions to data analysis and it is not great to get very different answers to the same question.

Any update to when these settings will be available?

I, also, need to make the answers more deterministic. My use case: Function calling (tools). When prompting the assistant to call a function, it currently will inject AI fabricated arguments, when I’ve clearly defined all the names, types, descriptions and instructions. I would love to be able to set the temperature and top_p.

3 Likes

Still waiting for update about this features.

Hello everyone, I’ll join my colleagues. It’s also necessary for our project to set the model temperature. Thank you for the opportunity to work with Assistants, and we look forward to updates.

I would like to reiterate what preston said. We have an application that uses the assistant api and need a lower temp and top_p to properly leverage the function calling feature.

It was mentioned that this would be included in your planning for the next beta update in November but I have not heard anymore updates.

+1 temperature would be very helpful for assistants using function calling

Really odd that such a basic configuration is missing. Let’s hope this gets exposed soon!

+1
This is really necessary. A lot of implementations require the highest possible accuracy in the answers.

+1 to this deterministic replies would be great.

+1 for temperature control, a must have for our applications.

I agree, we also need this feature to reduce variability in the assistant output.

Generally speaking, there have been very few updates to the Assistant API in the last few months, which makes me think that OpenAI has reduced the priority of this beta feature. Can someone from OpenAI please give a status update on Assistants? Are they still part of your plans? Your customers really need to know so we can plan appropriately. Thank you.

In the meantime, use the semantics of AssistantApi in the front and fulfill the request through chat completion in the back?

See Introducing openairetro -- AssistantAPI to ChatCompletion

We can use the OpenAI “Completion”, but we need use temperature in GPT4 API ASAP

Quick update here: we just added support for temperature.

5 Likes

Thanks, long overdue. Why not top_p?

Top-p, aka Nucleus sampling is more desirable: instead of merely reducing the chance of a token selection from the long tail of output-breaking undesirable tokens, top-p completely eliminates all but a most likely path, while still allowing diversity in ambiguous language positions.

top_p: 0.7, temperature: 1.7:

In the realm where technology thrives and blooms,
AI API sampling parameters loom,
Top-p and temperature, both crucial tools,
Guiding the algorithms, shaping the rules.

Nucleus sampling, a method refined,
Selecting the top probabilities to find,
Limiting the choices, yet leaving room,
For creativity to blossom and bloom.

Temperature, a variable in play,
Adjusting the randomness in a clever way,
Higher values bring chaos and surprise,
Lower ones lead to patterns and ties.

Together they dance in the AI’s domain,
Crafting narratives, sparking the brain,
Balancing order with chaos untamed,
In a symphony of data, they are named.

Top-p and temperature, partners in code,
Unleashing the power that lies untold,
In the heart of AI, where dreams take flight,
Guided by these parameters, shining bright.

1 Like

Awesome. I noticed it is documented in the API but not part of the published Python API yet?

2 Likes

I am also waiting for this basic feature should be added.

it is still disabled when I checked my Assistant. I am able to manually change the temperature but not getting saved. Can I use temperature as a parameter in the assistants method like so?
assistant = client.beta.assistants.create(
tools=[{“type”: “retrieval”}],
model=“gpt-3.5-turbo-1106”,
temperature=0,
file_ids=[file.id],
)