Is ChatGPT API actually getting worse?

[ruby_coder]

@RonaldGRuckus

To me, it makes no reasonable sense that the models in the playground output differently than the models that we make our calls

Yes, I understand your assumptions @RonaldGRuckus and you are not the only one who makes them here and elsewhere.

I also apologize for pointing this out so directly. I’m coding a different “non OpenAI” project as we speak, and am sure my reply comes across as “too blunt”, as they often do.

The OpenAI models are stochastic, so it is also not prudent to believe they will have deterministic outputs, based on input, especially for non 0 temperatures. The very nature of setting the temperature generates randomness.

So, if someone is using the Playground with a temperature of 0.7, for example, it is not correct to assume that an API call to the same endpoint by someone else will generate the same result (unless the model is overfitted, of course).

It’s easier to view the “non-overfitted” models as a kind of “cup of dice” and these stochastic models will generate a completion based on the temperature of the model (the randomness); so if you shake a cup of dice which are the same dice that I have, of course if we will get a different dice throw most of the time. This is true between consecutive API calls using the same code, based on temperature (randomness specified) and model fitting of course (not even considering the Playground).

However, with the Playground, we do not know how the input is finally filtered or moderated on the OpenAI side before the messages or prompt is sent to the API.

But honest, since I have written my own “Playground” which has a lot more features, I almost never use the Playground and have not attempted to reverse engineer it.

Anyway, I’m not trying to be a PITA, I’m just confident, based on writing two chatbots recently, using both the chat completion and the completion API methods, that there are myriad ways to filter, prune, summarize etc, the message history sent to a chatbot to maintain state; and slight variation in the implementation will change the output of the model.

Personally, I do not have the code in front of me on how the Playground does this, or how OpenAI may or may not add additional filtering to protect the OpenAI brand integrity, hence it’s hard to comment further without guessing.

However, my “best guess” is that OpenAI has some filtering and content moderation we are not directly aware of in the Playground because OpenAI must guard against people hacking around with prompts to generate headline grabbing compilations which will damage their brand integrity and mission.

Hope this helps.

[RonaldGRuckus]

No, I completely agree that the results aren’t the same by its nature.

However, to say that there’s inconsistencies between the API and playground, that cause playground to output better results is just wrong. The calls are practically the same is all I’m trying to say.

[ruby_coder]

@RonaldGRuckus

However, to say that there’s inconsistencies between the API and playground, that cause playground to output better results is just wrong.

No it is not wrong, for the many reasons I have already posted.

You are just assuming the API (which is not an application) and the Playground, which is an application are the same.

That is what is wrong @RonaldGRuckus

There is a lot of code written to maintain dialog state, which you would know if you sat down and wrote your own chatbot using the API to maintain dialog state.

Whatever you, as a developer, would come up with to manage state is not going to be the same as what the Playground does.

You are just making a huge assumption without any basic in fact; and that is why I keep calling you out on this.

It is easy for me, as a developer who has written two OpenAI chatbots which maintain dialog state, that the code on top of the API is the “secret sauce” (not the basic API calls).

The Playground is NOT the API. The Playground is an OpenAI software engineering developed application which uses the API; but there has to be code on top of the API (filtering, pruning, summarization, moderation, etc. what it is I do not know because I do not have the full application code in front of me). What happens between the UI and the rest of the process in the Playground happens with code written by OpenAI.

The Playground is NOT the API.

The Playground is an OpenAI application build on top of the API; just like any developer would develop a similar Playground, which I have written BTW; but my application is more more detailed than the Playground, and does a lot more (and of course, does it differently since I wrote the code and did not copy OpenAI’s code, as the Playground code is not public, open source to my knowledge).

If the Playground source code is public, open source, please post a like to it. According to the OpenAI GitHub official account, there is no Playground opens source:

Reference:

OpenAI

[RonaldGRuckus]

Yes, the playground is an interface so one doesn’t need to write their own code to try out the API. It uses the same endpoints.

To clarify, I know API and Playground aren’t comparable words. All I’m trying to say is that they make the same calls using the same parameters and endpoints that we have.

[ruby_coder]

@RonaldGRuckus

Yes, the playground is an interface so one doesn’t need to write their own code to try out the API. It uses the same endpoints.

There is a lot more to a chatbot application than the API endpoints.

In fact, the API calls are the most trivial part of building a chatbot application which maintains dialog and use session state.

My feeling, based on your replies @RonaldGRuckus is that you have not developed a a full blown OpenAI chatbot applications using the API; because if you had of actually written one, which had to manage session, state, pruning, summarization, etc then you would know that the API calls are the trivial part.

[RonaldGRuckus]

Playground by no means has anything to do with the actual processing or management. It’s a front-end service.

Yes, I have written them, and I’m fully aware of the complexities. The playground does no pruning, or context management. I don’t really understand what you mean by that. ChatGPT does, not the playground.

By definition, playground is a GUI

[ruby_coder]

@RonaldGRuckus

I don’t really understand what you mean by that. ChatGPT does, not the playground.

Sorry, @RonaldGRuckus

It’s not really a good use of our time to debate back and forth with you when. you continue to post your opinions and assumptions as fact.

@RonaldGRuckus

By definition, playground is a GUI

By whose definition?

Yours, of course @RonaldGRuckus

That is the core issue here. Whatever “you assume” and “you think” you offer as “hard facts”

I have written a Playground, and it is “much more than a UI” as you have stated @RonaldGRuckus, showing again, that you do not understand coding as much as you think. The API does not manage state. The Playground manages both user session and dialog state. It also may well add additional moderation and filtering to protect the OpenAI brand, as I keep saying but you keep rejecting.

OpenAI Lab: How I help OpenAI Community Members Here

Hi Everyone! First, I want to thank those who have contacted me here and on LinkedIn with all the kind words about how I help and assist members and users here! It’s very gratifying to be held in such high regard by so many developers here. THANK YOU. LinkedIn is the best place to contact me, as I have disabled direct messages here temporarily (except to a hand full off long time members, staff and regulars). So, what I have been doing with my current OpenAI project? Lately, I have been c…

I have not updated the topic above (busy coding), but it is more feature rich than before, including a
both chatbots, completion-api-based an chat-api-based:

I can assure you, @RonaldGRuckus that writing a full-blown chatbot app, playground or not, is not just writing UI code. The UI code is trivial (for a Rails developer like me).

Sorry, it’s pointless to continue this topic @RonaldGRuckus because you are wrong, but you think you are right and make so many assumptions based on “what you think” versus what is “a fact” that it’s not a good use of our time to continue, since I am almost sure next you will tell me you have written a “full blown playground”, blah blah, and you know all about that as well.

Take care.

[AgusPG]

I told myself not to answer in this topic anymore when I started seeing that we were getting off-topic and there was no real point on it. But I just needed to do it once again just to tell you that I admire your patience @ruby_coder. I truly do .

[ruby_coder]

@AgusPG

But I just needed to do it once again just to tell you that I admire your patience @ruby_coder. I truly do

I work out at the gym 5 days a week, which helps, believe me, haha

Thanks for the kinds words, @AgusPG

Like you, am done with this “full of misinformation” off-topic discussion. My wife wants to go for a drive in the country-side to a new coffee shop and that is “top priority” for the rest of today.

[RonaldGRuckus]

It’s fair that you don’t want to continue.

You’re clearly confusing a front-end service to a back-end one. Playground itself does not do anything that you are saying. Your front-end service doesn’t either. Playgrounds does not do any sort of business logic

My service is written in ReactJS, and then uses Kotlin with a sprinkle of Python in the back-end. Very hip, I know.

I would not say that my ReactJS web app does anything that you’re saying playground does. I would say that my back-end server manages of all that. Regardless, playground doesn’t even have these capabilities.

[Gaouzief]

we have an instance in France & another one in Central US,
same issue in both so it’s not location dependant,

this happened in previous weeks with text-davinci, same story repeating with chat apis…

[ruby_coder]

@AgusPG

But I just needed to do it once again just to tell you that I admire your patience @ruby_coder.

Here is my final post, illustrating there is an operational difference between the API and the Playground as well as an application-level difference, posting a screenshot just now from status.openai.com

Obviously, without looking into the nitty-gritty details, there is "relatively significant’ different between API outages and Playground outages. If the Playground was “the same” as the API, and “just a UI and nothing more” then of course the outage numbers between the API and the Playground app would line up nicely, but they do not.

Not only is the application layer software different (the Playground is an app built on top of OpenAI APIs with user session and user dialog management token management, perhaps OpenAI filtering and moderation, etc., the APIs are just API endpoints), but the operational context is difference, see below:

I promise @AgusPG this is now my last post in this topic

status.openai.com

[RonaldGRuckus]

Did you ever try to fix the problems?

I really don’t understand your patience comment. I’ve been spending my time to help your issue that you haven’t bothered to do any troubleshooting for, and then you pull the patience card? If all you wanted was to complain, just say so.

Believing that playground outputs differently is just plain wrong, and it’s essential to understand that before more people start to think “Playground somehow has better results (?)” When it doesn’t. The models in the playground are the same models that we have access to, there’s no differences. Even if playground had the fancy bells and whistles of context & token management ( it doesn’t ) it still doesn’t change the fact that the models and their output is the same. This is ridiculous

@ruby_coder your definitions are wrong. It’s as simple as that. Even ChatGPT is aware that the playground is a GUI. Of course it still needs a server to operate, I don’t understand your point. Even an HTML website needs a host such as apache to display. I’m seriously questioning your programming stack for not knowing what an interface is. Obviously the playground isn’t the API. It’s apples to oranges. It’s not even something to compare. My comments are saying that “Playground makes the exact same calls as we do”

[dennyroberts]

I’ve seen a bunch of posts since as early as January about the API and the Playground returning different results (and experienced this myself, frustrating). AFAIK no one has been able to explain or resolve the differences. The API seems more generic and less “in-character” (stuck in “helpful assistant” mode maybe?)

[PaulBellow]

It’s all about the prompt. I think my “grumpy, experienced Dungeon Master” did well…

AI DM with GPT-4 via ChatGPT: Dungeons and Dragons is About to Change Forever...

[RonaldGRuckus]

Until recently, the http request to the playground was the exact same as any other request. Now it looks like Davinci is called using a different endpoint, but ChatCompletions is the same (besides a different authorization key)

You can view it in the network tab. It’s not made by the server, it’s made by your computer.
In my experience here, the majority of discrepancies are caused by incorrect formatting or just plain wrong parameter settings…

[dennyroberts]

Interesting! Davinci was the one that was behaving strangely. I don’t think it would be incorrect formatting or wrong parameters in my case; I just copy/pasted the code it generates for my requests.

[ryan8]

I’m with you on the difference between playground and API.

Its very frustrating to use the same variables, same system message, same prompt, and get completely different results on the API versus the playground.

And gpt-turbo API will leave the role and switch speakers WAY easier than the playground. Watch how fast the API switches speakers on this app: LMS - HR Interview

[ruby_coder]

@RonaldGRuckus

@ruby_coder your definitions are wrong. It’s as simple as that.

Hi @RonaldGRuckus

You may have not noticed, but this is a community for software developers who actually write software using the OpenAI API.

I have read many of your posts and they are well written, well articulated but often (not always) what we used to say in engineering school “hand waving”.

You @RonaldGRuckus are intelligent and talk a lot by expressing an opinion, but according to a review of your history here, you have not written a line of your own OpenAI API code or a single application using the OpenAI API.

On the other hand, I have written three OpenAP API applications including one application which exercises the entire OpenAI API and is well know here as a “standard in helping others” here. I help users here with facts, actual tests for their benefit (at my expense), not opinions.

My “lab” applications has two, not one, fully operational “chatbot” apps, one which uses the completion API and one which uses the (new) chat completion API and has been used countless times here for it’s intended purpose, to help other developers.

In addition, that “lab” I coded, makes the Playground look like “Childs play” (based on functionality) compared to the Playground so for you to attempt to insult me, a guy who has coded every line of a “much bigger than the playground app” where you have not, is mind boggling.

The key discomfort that I have with your posts @RonaldGRuckus is that for someone who does has not demonstrated the experience in software engineering with the OpenAI API that many of us here have, including hand-on experience coding apps with the OpenAI API offering, your replies come across as knowledgeable and accurate when often they are not.

It’s kinda like ChatGPT, who provides elegant sounding replies which are technically flawed, but novices and “non coders” think that ChatGPT is helping them, when in fact ChatGPT is holding them back with misinformation; when they would have been better off to simply read the docs.

For folks like @AgusPG, @curt.kennedy and myself who actually know this stuff “inside and out” because we actually code it, some of us can easily see the “hand waving” nature of your replies and lack of depth of software engineering experience with the OpenAI API. The problem is that when we try to correct you, you begin go down the road of insults and offer up more of your “opinions” which are not based on fact, testing, or actual coding experience with the API.

Furthermore, when challenged, because you do not have this OpenAI API depth of coding experience (as demonstrated in your posts where you have not offered any code block you have written yourself or any application using the API which you actually coded, to my knowledge), and you do not post live tests to assist users where you actually preformed a test and posted the results to confirm your "theories, you edge toward insulting those you challenge you because you cannot offer valid proof of your “guesses” and “I think this” and “I think that” opinions.

So, if you want to start insulting me or @AgusPG because we, as real coders, do not agree with you, then go ahead; but if you continue, I am going to reply by posting exactly what I think of your misinformation posting here. We do not want to see novice coders here coding off misinformation and “guesses” and "well, maybe this is what I think… " We want facts based on hands-on coding with the OpenAP API, which appears to many of us that you do not have.

Yes, you can easily convince novice coders and beginners that you are very experienced, because you write well and are intelligent. However, you have not demonstrated, to me and others, that you actually have experience OpenAI AP coding with any degree of depth. You some across as a non-developer who has a lot of opinions, to be honest.

In my view, you are more like an intelligent, passionate retail consumer ChatGPT or Playground user, intelligent and passionate, who posts some, not all, misinformation because you have not actually written any software to any degree regarding the OpenAI API giving you true “depth of knowledge”.

I agree with everything @AgusPG has posted in reply to you and disagree with your replies to him; because I know for a fact that @AgusPG is a wise coder who actually codes with the OpenAI API now; and his posts are always full of good, solid information, where your posts are intelligently written but often laced with bits of misinformation here and their presented as fact.

You are arguing with two very experienced OpenAI API coders, @AgusPG and myself and because you are not on the right side of the facts, you are starting to go down the denigrate others, route. I advise you to stick with facts and not opinions and guesses and not go down the path of insulting others in your discussions.

Thanks.

[RonaldGRuckus]

I understand your viewpoint.

I am not a novice coder. I’m not going to display my trophies in a way to persuade you. There’s absolutely no way that what I’ve discussed has shown the opposite. I’m proficient in many languages, and am very passionate in what I do.

How you feel about me is perfectly fine. I have no issue with it. I hope that my future posts prove the opposite - although I would have thought that they already have.

I do have an issue with your persistence to be correct in a topic that doesn’t exist.
Playground models are the exact same models that we use. That has been the whole arguement.

You seem to think that I’m not aware that playground by itself is a whole level of coding. It’s not what I’m arguing and I don’t understand why you keep insisting that I am. The playground is a GUI to the models. This is not even comparable. The calls one makes is done on the computer, not the server. The models are the same.

Can we please kill this thread now.