My co-founder and I are building a tool to help plugin developers monitor their API and get valuable insights about their API usage with no effort.
So far, we are sticking to the documentation provided by the OpenAI team, and we have a working prototype (with a mocked ChatGPT client).
However, I would like to know if anyone here (especially a team member) would like to discuss our project. We would love to collaborate and we would like to be sure that what we’re building is not going against the OpenAI team’s vision.
If you are a developer and you are interested to discuss it, feel free to reach out too
In general, people use Mixpanel or Amplitude to track users’ behavior and create funnels, cohorts, and so on.
Even though the configuration is easy, it can be a hassle to add an event every time, especially when it comes to track successes and failures.
By using our tool, we act as a proxy between ChatGPT and your API.
This means you can track all successes and all errors over time without any additional effort. You will be able to know in what circumstances ChatGPT has decided to call your API and with what parameters. You will also get a precise idea of your API health for your ChatGPT plugin features.
We also believe that as the ecosystem grows, we will be able to provide more and more value. For instance, by providing versioning for your Manifest, simplifying how authentication will work, or more.
Feel free to share your thoughts, this would be really helpful
Cool startup idea with lots of potentials. On the surface this might sound like the “Datadog+Google Analytics for plugin developers”, but the real value, I imagine, is “Ahrefs for ChatGPT Plugins”.
I’d be down to chat about that part.
Given the plugin architecture today, your gateway product will be one of these:
an SDK/API
an API proxy
a low-code hosting platform
a browser extension
Going to require a bit of setup regardless. My 2 cents: it’s hard to justify “with no effort”. Plus, “no effort” also means “easy to replace” - a double edge sword that you probably don’t want. I’d focus on providing insights rather than just build another dev tool.
Given the plugin architecture today, your gateway product will be one of these:
an SDK/API
an API proxy
a low-code hosting platform
a browser extension
Nice guess! We are implementing the second option as a proxy between ChatGPT and your API.
it’s hard to justify “with no effort”
To be fair, it requires some dev skills. But the only configuration that is needed is to add an A entry at your DNS level and to update your manifest according to the URL we provide you.
It takes literally a few minutes to get started VS setting many events manually on your backend.
also means “easy to replace” - a double edge sword that you probably don’t want.
That’s a good point! Thanks for sharing
I’d focus on providing insights rather than just build another dev tool.
Interesting! Could you elaborate? What I understand is you’re suggesting to focusing more on “tracking user’s behaviors” than “monitoring a backend for developers”. Or put another way : focus on business rather than on the technical level. Is that what you meant?
interesting idea. For me, in the moonshot case the plugin gets picked up, and there’s the ability to do paid/subscription-based models, the biggest value add would be determining the avg cost/user of api calls per month. Would be great information for pricing, determining any potential budgeting.
Given the plugin architecture today, your gateway product will be one of these:
That’s a right guess. We are actually doing the option 2 ; acting as a proxy between ChatGPT and your API.
it’s hard to justify “with no effort” .
That’s a fair point. Actually you’ll need to create DNS level A entry that points to our servers.
Once that’s done you’ll have to update your manifest with the correct URL, et voilà.
It’s fair to say it requires some effort, although the whole process takes literally a few minutes.
On the other hand, you not have any additional configuration compared to Mixpanel or such a tool where you always have to manually add events.
. Plus, “no effort” also means “easy to replace” -
That’s a good point. I will keep in mind for sure!
I’d focus on providing insights rather than just build another dev tool.
Interesting! So basically what you mean is it would provide more value to focus on user behavior and funnels than on the technical monitoring part for tech people. Is that correct?
Just to clarify, do you mean that our tool could help plugin creators to set up subscriptions/paid models and determine the usage/cost of their plugin?
Or do you mean we could create our own subscription system for people who use our solution?
I’m trying to grasp the proposed value addition. Let me outline my instrumentation and perhaps you can clarify the benefits your service offers.
Your current setup includes:
Tracking install events to monitor total installs and differentiate new vs. returning users.
Capturing error events for each route and handler.
Monitoring critical flow events, like friendship actions.
Logging issue and feature request events.
Recording all API usage.
Sentry handles these tasks, and the data can be exported to monitoring tools like Grafana. As for logging, you utilize Papertrail for:
API usage.
Successful operations (happy paths).
System health.
Considering this, I’m curious about the unique advantages your service brings compared to the existing industry standards for production applications. I’m open to integrating your services into our products, but it would be helpful to understand how they stand out from the competition.
Thank you for asking this question. It’s helpful, especially at our stage, to get early feedback
That’s correct; You can use either Sentry (to monitor APIs) or Amplitude (more for user insights).
Our take is that these tools are not focused at all on the Plugins ecosystem.
What we are currently building is more of a toolbelt for plugin creators who wanna launch fast and get additional features.
Our tool will be 100% designed around the plugin ecosystem.
You will be able to set up plugins, generate manifests, manage their versions, and more.
Let’s be honest, we are actively building it, so the main value proposition is changing as we talk to potential users.
Short-term our product is not going to suit enterprise-grade needs but will be more adapted to plugin creators who would like to launch quickly.
However, acting as a middleman between ChatGPT and your plugin opens tons of opportunities.
Actually, we are currently moving toward a slightly different value proposition: providing the most effortless payment solution built entirely around plugins.
It would work as follow:
→ Link your stripe account
→ Create your plans with a quota on our platform
→ We handle rate limits and payment gateway for you
This would probably not fit your use case at all but I would love to get your thoughts if you get a chance
I understand where you’re coming from. As the world embraces low-code development, simplifying these processes will indeed be beneficial to many. While I personally find the tasks you mentioned to be straightforward and easy to set up, I acknowledge that it’s not the case for everyone, and even those who can may not want to invest the time.
Your efforts will undoubtedly be advantageous for the influx of new developers learning and building in this space. Though your current proposed plan may not appeal as much to experienced developers, they likely aren’t your primary target audience at this stage anyway.
I’m looking forward to seeing your Stripe integration. It’ll be crucial to provide comprehensive documentation and onboarding, especially if you’re targeting beginner developers. You’ll need to guide them through handling production vs test environments, webhooks, product vs price IDs, subscriptions vs single payments, and so on.
Feel free to reach out if there’s any way I can assist you. Good luck!
Hi All, I think this is a great idea, and the plugins environment may be indeed underserved now. I am also developing an llm-specific analytics platform, as I wanted a way to be able to block users from spamming calls on a web app I was building! The platform allows you to see avg API calls and cost per user, avg tokens used per user etc. I also am developing a way to detect prompt injection attemps, even though LLM-app security is very poorly understood and implemented (even by Bing!) even rudimentary features can help low-resource developers such as myself! Feel free to see more info at llmetrics.app
I’m probably late to the party, but that may be a good thing by now. I’m curious, what is your tool? Will there be a beta version? Are you looking help on the project? I am especially interested in this.
This seems like a common, basic as in “People are already doing it” kind of idea. Now i support you in an idea assisting others, but… this seems too general. Its something people already do. Even i know that, and i went from knowing nothing about programming at all, to whatever i am now in the span of being granted plugin access to GPT4… perhaps thats something to know.
