Different results from openai and playground

Hi,

I’m sending in a completion request for a Q&A. I’m providing an email that we got and asking what the revenue of the company mentioned is. I am using ‘text-babbage-001’.
In the playground I get the correct answer, but when using the api it randomly adds 811,000 to the amount.
Also if the revenue isn’t mentioned it return 811,000 as the amount.
Any ideas why?
when I use curie, it seems to be ok (for a specific example - I didn’t run it on the whole set).

Thanks!

4 Likes

@at-bay-ds-team I’m curious if you found a solution since I have the same problem. I literally copy the prompt string and paste it in the playground and get a (consistent) answer. My local code gets a (consistent) different answer.

I have gone so far and used the chrome developer tools to look at the settings that are used in the http post and made sure they match with no effect.

I understand that there is some inherent randomness and that answers aren’t always the same, but this seems like something else is going on.

2 Likes

Babbage is not very good to get answers from embeddings. I would stick with DaVinci. A few other things:

  1. What prompt are you giving to the bot? You need to say that "if a question can’t be answered based on the knowledge base, respond with “Unknown” or something like that.
  2. What temperature are you using? Set it to zero.
  3. Make sure you are using the eight API to create embedding for the knowedge base and question respectively. These are two different engines.
2 Likes

It doesn’t seem to me that there is an issue of randomness here. I am consistently getting the same answer in the playground, and consistently getting the same wrong answer through the api.

babbage is giving the right answer on the playground - and I am using the code provided there.
At this point I will have to use a different model, but the question is why is the difference occurring?

this is the code I am using:

response = openai.Completion.create(
            model=openai_model,
            prompt=f"{text}\nQ: what was the revenue for the last fiscal year?\n\nA:",
            temperature=0,
            max_tokens=100,
            top_p=1,
            frequency_penalty=0,
            presence_penalty=0,
            stop=["\n"]
        )

temperature is 0.
I am new to openai - how would you implement your suggestions (points 1 and 3)?

Try doing this. Whatever question a user is asking, append that question with "If you can’t find the answer in the provided context, say “unknown”. Let me know how tat goes.

I should have clarified what I was using! I am using the completion API from Python and these settings:

        model="text-davinci-002",
        prompt=prompt,
        temperature=0,
        max_tokens=256,
        top_p=1,
        frequency_penalty=0,
        presence_penalty=0,
        best_of=1,
        stop=["\n"]

It always gives an answer that is right-ish, but just different so it isn’t a problem with not knowing.

I wonder if the playground gets “newer” or “test builds” before they roll out?

Just try changing the prompt as I suggested and let me know.

I tried this - but then it just gave ‘unknown’ for everything

I tried this with a different prompt using davinci and it works. :smile:
This still doesn’t explain the discrepancy between the playground and the api…

2 Likes