Hey folks! I’m one of the engineers working on the plugin developer experience at OpenAI. We’re exploring better visibility into plugin analytics, and I’d love your input.
What’s missing for you today: usage stats, tool errors, latency, user feedback, something else? Share what you’d want to see. Concrete examples are especially useful
Hi Henry 
,
Thank you for reaching out. Your question comes at a particularly relevant time for me.
On Saturday, September 19, I published a topic on the OpenAI Developer Community titled “Custom GPT → Plugins: 100-scenario baseline”, explaining how I established a 100-scenario non-regression benchmark for my Custom GPT, GM INFO - IA Technologies ©, before migrating it to the Plugin architecture.
In this context, the analytics I would find most valuable are those that make it possible to clearly distinguish a change in model behavior from an issue in Plugin execution: active users and invocation counts, per-tool statistics, success and failure rates, error categories or codes, timeouts and retries, authentication or permission failures, as well as latency metrics such as P50/P95/P99.
It would also be extremely useful to associate those metrics with a specific Plugin version and to compare two versions or two time periods. For developers migrating a mature Custom GPT, this would make genuine regression analysis possible after each change.
A concrete example from my own project: after migrating GM INFO - IA Technologies ©, I intend to rerun exactly the same 100-scenario benchmark. If one scenario regresses, being able to determine whether the expected tool was actually invoked, whether it succeeded or failed, its latency, and the relevant error category would make the analysis much more precise and actionable.
Explicit user feedback would also be valuable when it can be associated, in an aggregated and privacy-preserving way, with a particular version or invocation.
From my perspective, one of the most useful capabilities would therefore be versioned and exportable analytics, through either a dashboard or an API, allowing developers to objectively compare two Plugin versions.
I would naturally prefer aggregated and anonymized metrics by default, without exposing users’ conversation or prompt content.
Thank you for actively seeking feedback from builders. I would also be happy to share my post-migration comparison and delta analysis if that would be useful to your team.
Kind regards,
Gregory Mayor
GM INFO - IT SERVICE DESK & GM INFO - IA TECHNOLOGIES
There is one aspect in metrics that I think should also be added from the aspect of usability and accomplishments done by the model used. Usually most of the user feedback is done by a thumbs up or thumbs down, and that is a very vague metric. Instead there should be a more tailored feedback/gamification aspect to it in which workflows for the plugin in question can be asked to the user.
Was the intended workflow accomplished? Was the output what the user requested by using the plugin?
I made a suggestion for feedback a while ago here and this could be tailored to the Plugin metrics.
Hope this gives an idea.
Cheers!
Dys
Thanks for pinging @VeitB 
@henry-leibenko, I’d say most “basic” metrics, like tool calls, latency, input categorization… can already be collected using 3rd party solution (especially if you need to aggregate across multiple host where the plugin is used).
The one I’d love to see first in my OpenAI platform back-office are the one I can’t have any other way. For example, the number of installation (given the fact all installation do not trigger a tool call necessarily) is a metric I’d be interested in, and how I rank compared to other plugin in the same category on that metric.
I’d also be very interested in demographics of user installing/using my plugin (anonymous id is the only available information so far, great to measure retention, but can’t tell much more).
Would you consider making such metrics available?
Thanks!
Agree - I’d focus on the stuff we can only get from OAI as the platform vendor.
I’d love visibility into the full plugin/tool funnel, not just total usage.
For example:
User asks for something my plugin can handle → was the plugin available? → was it considered/selected? → which tool was called? → did auth/permissions block it? → did the tool error or time out? → was it retried? → did ChatGPT fall back to another path? → did the user actually get a successful result?
A few metrics/views that would be especially useful:
Concrete example: if usage suddenly drops, I’d want to know whether users stopped asking for the feature, ChatGPT stopped selecting my plugin, authorization started failing, or my tool itself broke. Those are four completely different problems, and aggregate call counts wouldn’t tell me which one happened.
Some sort of version-aware dashboard would also be huge so we can correlate a schema/description change with selection rate, success rate, and latency.
I’m building a B2B MCP server, and these are the three things on my wishlist. The last one is more about transparency than metrics:
+1 to “did the user actually finish what they came to do?” You can get a decent approximation just from the tool calls, without a judge model: simple rules for completed, retried and succeeded, stuck in a loop, or dropped off after an error. I already do this server-side. What I can’t see is when someone asks for something and the model never calls my tools. Only the platform can show that, and it would help me tell whether my tool failed or never got considered.
Give us before/after numbers per tool. “78% success” doesn’t tell me much. “Argument errors on create_order dropped from 12% to 3% after Tuesday’s description update” gives me something to work with. Let us mark when we change something and compare the metrics across versions.
Show us what the model actually sees. Someone in another thread mentioned that only roughly the first 512 characters of server instructions reliably reach the model. If that’s the case, I’d really want to know. Truncating, summarizing, or reordering our metadata can change how the agent behaves, and right now we’re left guessing. A view of our tool definitions exactly as the model receives them would save a lot of debugging.
Happy to share the session-classification rules if useful!!
One thing I’d really like OpenAI to consider is a user-facing RAG / knowledge inspector for connected apps and user-owned data.
I actually had this need before: I wanted a way to inspect the mapping behind my own RAG layer — not the model’s internal reasoning, but the user-owned sources that ChatGPT is allowed to retrieve from.
Ideally, users could open an app/plugin and see something like:
Connected source → document → chunk/index → topic/relation → retrieval → answer
Useful views could include:
which files/sources are currently indexed
document/chunk mappings
metadata and freshness / last indexed time
topic or relationship mapping between documents
which chunks were retrieved for a specific question
retrieval ranking/score
stale, duplicate, missing, or excluded content
permission boundaries across connected apps
a manual “test retrieval” mode where a user enters a query and sees what the system would retrieve
It doesn’t need to expose raw embeddings or internal model reasoning. A visual knowledge map / retrieval trace would already be extremely useful.
This would help answer a very practical question:
“What does ChatGPT currently know from my connected data, and why did it retrieve this source instead of another one?”
I think this could be especially valuable as an OpenAI-provided plugin/app, so users can inspect the knowledge layer across the apps they have connected rather than every plugin developer building their own observability UI.
For advanced users, I could imagine an “index explorer” view similar to:
Query → candidate chunks → selected chunks → source documents → model response
That would make debugging stale retrieval, missing context, wrong document selection, and connector permission issues much easier.