Lack of determinisim even with temp 0 and fixed seed

Hi there,

We are having a hard time getting reproducible results across the board even when the seed is fixed and when the temp is set to zero. Is this expected? The system fingerprint output is also None for our calls, which doesnt make a lot of sense given that this would be the recommended way of tracking the system state at least according to the api docs. https://platform.openai.com/docs/api-reference/chat/object

Only models since 2023-11-06 have a system fingerprint returned. The last I checked, they were all disabled or not working except for one model.

You can see from reading the rest of that topic, they have little value now or ever.

That “determinism is not guaranteed” = it should have never even been offered because OpenAI also offers “flawed” models that do not do the same computations nor return the same logprob values between runs.

Temperature:0 is a short-circuit to a value that is not as low as 1e-6 or smaller that you could specify (smaller values are now broken on gpt-4o).

The best parameter to only return the top “deterministic” token is top_p:0.000001, which could only return the top token, but then the only research you can do is how often the top token alternates ranks with a second-place token between identical runs.

There is little need for actual reproducible results in practice.