Love this! And thanks for releasing the github reference implementation.
I am glad that we are already working on a lot of the “Future” work (55+ document types, Youtube/podcast transcriptions, small business-friendly User interfaces, advanced embedding calculations, chunk relevancy, etc)
Can’t wait to build the plugin and integrate our capabilities into ChatGPT.
This is truly awesome. The more I read about it, the more thrilled I am. For me, this is the most disrupting event in the last months. Even more important than ChatGPT API or GPT-4 (at least, in my view).
Just wanted to address the elephant in the room here. Maybe these questions are answered somewhere else. But in case they are not:
Great questions.
It would be completely ignorant to not think that this will become another monetized app store with SEO and other technique$ to push competitors to the top. It’s a very strange move to only focus these plugins in ChatGPT. It makes me feel like they are aiming to make their product completely centralized.
They keep bundling “developers!” and “ChatGPT” together like we have some sort of control over it. As of right now, I can do absolutely nothing with ChatGPT, so why are they talking like I can? Is this indicating that they are shifting from open models to ChatGPT with plugins? I have no clue because they don’t tell us anything!
These updates come as a constant surprise (good surprises, just frustrating because it’s now been multiple times I’m working on improving something, for it to become wasted time as there’s no roadmaps or slight indications on what is being worked on).
So now I’m left completely confused, and as a small developer I am lost in resources, time, and direction. I have no idea how I am supposed to adjust for this update. Should I continue with my version of plugins? Or should I wait for them to become available to other models? Is ChatGPT going be the focused unit for commercialization? It’s a huge shame because it seems like big companies know these answers, and small devs are left wondering.
It’s still an amazing step forward, and I completely agree that this is a huge update. Just a bit concerning. OpenAI is moving so fast, I feel like I’ve been left in a cloud of dust, just trying my best to figure out the right direction to go.
Would appreciate clarification on a few points:
Is it the case that every API description for every plug-in activated by a user is sent to the model on every query, taking up limited context? If so, it seems like that will quickly cause the context to fill up. If not, how is the list of APIs that will be considered for each call determined?
It seems API calls are made into API plug-ins from the browser (rather than from OpenAI directly.) However, this would mean that the API key needs to pass through the browser. If that’s the case does that put the key at risk of being exposed?
To second @RonaldGRuckus. I start to feel like OpenAI is a landmine for developers. You can be working on something only for them to release it next. It feels like they are not so much interested in an ecosystem than doing everything themselves in their centralized model. It feels like the Steve Balmer’s days at Microsoft almost.
I can see that this is a major step forward and adds an enormous amount to the scope of the GPT models, but doesn’t it also represent something of a conceptual shift and therefore challenge?
Until now the GPT models have been “self-contained” and that has in some respects seemed like a limitation, but it has had a significant advantage as well inasmuch as the “answers”/“completions” provided by the models have always been generated internally, and so could range over the whole scope of the training data, almost certainly in ways no human mind could understand (the black box problem).
Once we incorporate plug-ins and third-party APIs this ceases to be the case, or at least cease to be exclusively the case. Unless I have misunderstood the announcement and Stephen Wolfram’s blogpost, the LLMs are now drawing on external agencies that they have not been trained on, and so cannot possibly integrate into their processing in the same way as their training-data. It’s exactly the analogy of “RAM - v - Hard Drive” or in human terms “Knowing it” - v - “Knowing where I can find it”: the difference between “having” knowledge and understanding and “referring to it” is profound in terms of what our nonconscious brains - neural nets, don’t ya know - can take into account in an integrated way in our “thinking”.
So “modified rapture”. 
I don’t believe the intention of the plugins is to fill in on missing general knowledge, but to connect to other API services which send out service & product information. So for example, today’s flight information, restaurant prices, product specifications, sport’s updates. etc.
Actually, I take it somewhat back. It looks like one of the endpoints is a … legal directory?? Yikes… That seems like a disaster waiting to happen. What if ChatGPT disagrees or hallucinates information from it?
Thank you. Agreed. Sure, never in doubt. But the incorporation of Wolfram technology naturally invites the question whether it will also bring better mathematical reasoning into the frame, which is currently a notable limitation as OpenAI have long-recognised. Hence my point: unless some post-LLM model “absorbs” the kind of capacity Wolfram’s Mathematica embodies, it’s hard to see how the internal process moves forward at least in that regard.
That’s a very good point.
It may now be able to solve very complex mathematics, but it may not understand “how” it solved it as it passed that responsibility over - which I would imagine would result in hallucinations. Unless it truly does understand the process and just cannot simply do it, or perhaps Wolfram explains it, and is omitted.
it will be fun to test that theory out.
Thank you, again. I am very excited by this. It’s potentially a game-changer. It’s the “What do you do when you don’t know how to go on?” problem: Wolfram’s Mathematica is a fantastic “tool”, but it doesn’t pretend to be “intelligent” or to have problem-solving abilities; it relies on users to provide those.
I think the GPT family do have problem-solving abilities for exactly the reason that they draw on untold networks of connections - their ability to extract patterns from unimaginable quantities of data - that neither they nor we can pretend to understand, to produce their “output”. This, it seems to me, is exactly what great mathematicians can do: take a problem and somehow conjure up a solution from the fragments of existing knowledge.
If you can synthesise like GPT and know how like Mathematica, … wow! 
The LLMs are not doing this. The ChatGPT client is asking the LLM to formulate a third party query plan, and then the client is using context obtained from that third party to generate a response. This is basically data augmented generation using the ReAct pattern, which is not new. It is what libraries like LangChain help you do with custom apps. Now, OpenAI has embedded this capability in their own client. The philosophy of how the LLM fits into the solutions has not changed. All that’s changed is OpenAI is moving up the stack a bit.
Thank you. I appreciate your response. I think we agree: that is old technology, and while it may greatly increase the scope of the GPT responses, current utility, and so profitability, it isn’t a big step forward.
But that’s not my point: what the LLMs are doing even now is far beyond mere “referencing”: it’s creating new content. So why not create new mathematical content? Because mathematical problem-solving isn’t just about “completion”. It isn’t a simple “completion” scenario: it’s an integrated understanding scenario.
ChatPDF works with PDFs. You upload and basically “chat” with the PDF, so its information only comes from the PDF, not from the GPT database.
Understood.
However, it seems like soon ChatGPT in itself will be able to read and discuss PDF documents without a third-party application such as ChatPDF - which is what I was referencing.
Someone (such as the ChatPDF dev) can be working very hard on their technology. OpenAI creates the technology, or changes the direction and completely negates what the developer has been working on. The big businesses most likely are given a roadmap, as they have these “plugins” already developed before we are even aware of them. So the smaller developers are not only left at a huge disadvantage, but also risk their technology being completely shadowed and rendered useless.
As a small developer trying to incorporate this technology, it has been completely exhausting.
It comes to a point of asking “How do I know that this won’t become redundant in a couple weeks?” We are left essentially guessing what the future of OpenAI’s products are, and how we can become a part of it because there is no communication. It would be understandable because of how fast the technology is growing, however it’s not understandable because we are purposely left not knowing the direction of growth.
It seems like once all the dust has cleared, the potential markets will be swallowed whole by the businesses that had the insider knowledge to prepare for them.
It really sucks. Because, for example, the ChatPDF developer could have been focusing on creating the plugin, and made their space in the ChatGPT plugin store. Instead, it looks like big companies get first dibs, and then we are left waiting to not only be aware of its existence, but also to even be able to try it out.
Yay capitalism! 
I feel what you’re saying mate, 100%. A road map that’s open for all to see sure would be helpful.
I imagine many software developers are feeling a bit like the candle maker.
I can’t help but feel that the next step for these plugins is that ChatGPT will just create them for you. “Site A has an API. I would like to be able to do B, C, and D. Use the api and my key to create a plugin in Python.”
I never heard of the candle maker story. Great reference.
I wouldn’t be surprised if GPT-4 is already capable of doing this.
Verbally defined pipelines. What a thought.
A neat YouTube video with David Orban and Stephen Wolfram that just came out.
ChatGPT, AI, and AGI with Stephen Wolfram https://www.youtube.com/live/szxiPMyuMGY via @davidorban @stephen_wolfram
Guys, it has been super exhausting, but look at the opportunities!
99% of people are ignorant to the “API/plugin/OpenAI power”.
We don’t need to invent google to be successful, focus on a use case and commit to it. There’s no sense in chasing tails or trying to create the perfect product.
Still, some public roadmap would be excellent.
Would there be a way to make a template for plugins that has stripe/venmo/cashapp/paypal/applepay prebuilt on vercel with the openai api setup.
This would enable developers to charge pay-per-use for extremely intricate and helpful plugins, and acknowledge apps are going to come and go so let the devs get paid and incentivized to make more stuff which will accelerate the types of things we build as people can go straight to sophisticated uses with a true template that can help fund the dev.
Pay per use minimizes up front cost to users who may hesitate to buy credits or sign up for a monthly subscription. More use = faster evolution toward seeing what these multi prompts and new forms of access can really do.