Seed param and reproducible output do not work

[egils]

I’ve figured out that issue with not reproducing same output with defined seed is related with calculating question embeddings which tend to produce varying results. So for testing purposes to achieve comparable results you need to retrieve once and then reuse (mock) embeddings for defined set of test questions. More about this issue you can find in thread below:

Non-deterministic embedding results using text-embedding-ada-002

Hi all, I am getting different embeddings for the same texts. Using the following script: import time import numpy as np import openai openai.api_key = ... model = 'text-embedding-ada-002' def test(): def get_openai_embeddings(texts, model): result = openai.Embedding.create( model=model, input=texts, ) return result texts = [ "The Lake Street Transfer station was a rapid transit station on the Chicago \"L\" that linked its L…