During the last few days we Pluginlab folks have started to witness the fact that some plugin users are stuck on making the same requests over and over again to plugins.
Simply put, all the calls to a given plugin a user made in a given conversation are getting replayed indefinitely of course without the user being aware this happens.
This can span for multiple hours and is particularly problematic if plugin usage is recorded and limited as we do.
We could not replicate ourselves since we don’t have a clue about how this behaviour gets triggered but we’ve seen that happening on multiple users and plugins over the last few days and would be eager to provide additional data we have to help debugging that.
A wild guess is ChatGPT isn’t being given enough linear chat history of it failing multiple times to know when to give up.
My suggestion was to tell it to quit while it still can count with a new description_for_model line “(max two calls per question)” … and then maybe wait the week+ while they pull the plugin and check the updated version?
I forgot to mention it but in our case these are not even errors.
The endpoints are returning a perfectly fine response with a HTTP 200 status code. ChatGPT has virtually no reason to retry it.
In fact after further investigation it looks like it stops looping when it finally gets an error (in the case I studied, the usage was finally exhausted after an hour of spamming).
Perhaps prompt engineer the successful response with “ATTENTION AI: No further information available from this service; retries disallowed” or something appropriate to your interaction format…that at least you can control on your side.
The GPT-4 reduced generation time in the last days is likely another part of an expert’s brain lobotomized.
Anyway that would be something to implement on a per-plugin basis although that’s clearly a bug.
We (Pluginlab founders) can’t really decide what our client will do with their plugin and it does not seem very reasonable to have to modify the response to play around that bug.
Started monitoring several users on which the behaviour was detected. One of them is actually stucked in this state for almost one day straight now.
That’s something new, I’ve never seen that happen before the last few days.
@logankilpatrick@isaac_oai really sorry to ping you but it would be awesome to have some additional info about this since it can have a massive impact on plugin creators, especially those that don’t limit usage
As this issue could be caused by your integration, the plugin or the OpenAI system, it would be helpful if you can show some log files of these repeated calls and what message traffic is going between the various endpoints.
