I got very inconsistent result when testing in Playground, below is my scenario, and also I put my Assistant ID in here in case it helps OpenAI team to troubleshoot.
Assistant ID: asst_Gr9kmM3oJGODWViLvZ8Eu41J
Same question on all 5 runs:
“What’s the latest recommendation to treat COVID?”
Model: gpt-4-1106-preview
3 Files attached all texts
Run 1 → thread_id: thread_9SwHQkqzxg0rlvrdTqmCqOv6 | run_id: run_iJ643Ma0pSMcmizSSipEDm7k → still running
Run 2 → thread_id: thread_dFedgAD8HGymHSabjo8v1mBM | run_id: run_1JvPP3ewU8YTP9wifEJ2x2Eo → still running
Run 3 → thread_id: thread_2uoFn9c1uCi7LuHA0AszvKrB | run_id: run_tJC22hunGqFkgwevS2lGBaIo → answer returned within 10 seconds
Run 4 → thread_id: thread_G6MrDmJjdeP9BDB6krxU8g29 | run_id: run_Z72xkDzW5H06RnLIs4JcTRcD → answer returned within 10 seconds
Run 5 → thread_id: thread_8SpfPKYtgaDIrcKI6AijVJjR | run_id: run_7pGf0fAJuJNn7nKLk8uf5sxp → still running
Any ideas why my results are very inconsistent? Maybe some indexing issue on the OpenAI side?
It should be noted that its knowledge cutoff is going to be April 2023, so I hope you’re using web results to accommodate this, and that these runs exemplify successes of those website calls.
The problem you’re facing is that AI responses are meant to have variance. That is not going to go away. What we can do though is attempt to reduce the amount of inaccurate responses, but that will not be 100% every time.
Thanks for the reply. But I am asking Assistant AI to respond based on 3 files that I provided. As a matter of fact, I am using Assistant AI specifically because I don’t want OpenAI to go outside the files I provide, to answer the question I have.
The problem I have is not with the variance in the response. The problem is that, if I ask the same question 5 times, it won’t even return with any response 3 out of those 5 times.
It feels like the Assistant AI system is “stuck” 60% of the time.