What triggers an app re-verify?

I’m looking for an official answer, or answer from developer experience.

For all scenarios, assume it doesn’t substantially change the purpose of the app.

  1. Modifying the manifest file = triggers a re-verify. Confirmed.

  2. Adding a new /endpoint in OpenAPI spec = Unknown

  3. Modifying an /endpoint description in OpenAPI spec = Unknown

  4. Modifying the output of an /endpoint = Safe.

4 Likes

Related thread: https://community.openai.com/t/plugin-knocked-down-to-unverified-status-without-any-changes/220709/3

“Our code is very much looking for a change in the manifest file.”

As developers, we’re trying to understand how to improve the user experience, safely.

The link is 404 for me.

Are you saying once the plugin is submitted, ChatGPT services poll the manifest for changes? Then some type of verification is done by the ChatGPT team and then pushed to users?

So there is no way to make changes to the manifest for testing purposes w/o it potentially being available to live users?

We got pulled off the store for 4 days for re-verification due to a change in manifest. I confirm that this is not safe to change once you’re live.

However, the other scenarios are not clear to me yet, and I hope to gain some insights!

1 Like

Right, it would be nice to get clarity around whether a change in the OAS file would cause a re-verification since this impacts rollouts.

Heres the issues I see

  • If they don’t poll the OAS file, then we would need to modify the manifest in some trivial way for spec changes to pick up. If this is the case, they should add a version field to the manifest.
  • If they auto-verify, w/o allowing dev to approve final release, then we have to support two versions of the backend API between the time we change the manifest and the time that they approve the changes. That would be a pain.

Right now, it is looking at the manifest file (ai-plugin.json) for changes, and will auto delist if the version it fetches is different than the version we have cached. This will likely extend to other parts of the system in the coming days / weeks.

6 Likes

I found out the hard way :slight_smile:

What is best practice for releasing a new version?

1 Like

oops. found it

Right now, it is just me doing plugin reviews, along with 100 other things, trying to scale this process as much as possible, honest suggestion is minimize changes to the ai-plugin file unless you want some amount of downtime.

10 Likes

So on manifest change, its delisted from the plugin store. Can current users can still use the plugin? Or is the plugin also disabled on manifest change?

1 Like

The current users can still use the plugin.
The plugin is not disabled. It only affects new users.

1 Like

From watching my server logs, my chess plugin continued to be used even though it switched to unverified.

But obviously no one could install it in the unverified state.

1 Like

In this case, since devs don’t have full control over the timing of their release, they will need to maintain two different versions of the API endpoints when there are breaking changes.

1 Like

This is all very bleeding edge. Almost surprised it works as well as it does.
I’m sure the experience will get better.

My guess (not official) is that this has to do with global caching and how that call maps from the transformer to your plugin.

In the meantime, I highly recommend revisions as long as the plugin yaml changes.

Separately, I realized that simpler api is better. I have a lot of knobs but no one uses them.
So your plugin yaml shouldn’t change by much.

1 Like

Hi @logankilpatrick,
You might be doing this already, but I was wondering if we could utilize the capabilities of GPT to conduct plugin reviews. This way, you could pre-filter the plugins that don’t meet OpenAI standards, allowing you to concentrate on the ones that could potentially be useful. :grinning:

@logankilpatrick When a regular plugin becomes unverified, existing users can keep using it until re-verification. However, when a plugin with OAuth becomes unverified, OpenAI stops renewing tokens, so the app becomes completely unavailable to existing users. What are the plans to address this?

We are currently holding back from improving the plugin based on user experience, because it would possibly result in being pulled from the store.

To be clear, we don’t plan to update manifest. Is it safe to update the OpenAPI spec only?