As for the magic of the API calls from sniffing on the manifest, it will have to be sophisticated to actually work in a general way. ChatGPT is actually not so good at generating SQL.
Sure we could have used a better prompt and more advanced model. Still…
Good question. I would definitely be happy with that.
I agree with all that you have said. I truly believe that in n years we will speak our ways through the days.
Order groceries in the kitchen while we prepare ingredients, plan vacations while cutting vegetables, and then have an automatically generated schedule for the rest of the day.
Or even just order chinese take-out with a single “My favorite, delivered in an hour please” said from the living room couch.
None of this technology is ready though. ChatGPT plugins will be massive. It’s an app store. I can understand the reason for it, I just want OpenAI to focus on their dang models, and not focus on complete domination of the inevitable AI-entwined future.
Stay on one path, and let others take advantage of the technology to venture off, and create separate paths. Ya know. I’m sorry, this doesn’t have much to do anymore with your discussion.
Going back, text-to-sql is the hot topic it seems lately. I have been using an ada to perform simple filtering for my database retrievals (using vector databases though) and it’s working very well.
Indeed. And they will generally go there for general tasks. Increasingly, consumers and businesses will switch from Google search to ChatGPT for various answers and helpful completions, conversations, and content development.
It’s a silly notion, of course. But why didn’t search plugins evolve to do this if they are such a great search engine? The public is vastly hooked on semantic search but apparently not enough such that they wouldn’t demand direct access to their private or corporate data to use the same UI to access everything
The answer is obvious - it’s a weirdly stupid idea fraught with many serious issues, the worst of which is trust.
Plugins will be massively successful, but probably not for the use cases you’re suggesting.
The first rule of plugin access: don’t talk about plugin access. 
The first rule of plugin access: don’t talk about plugin access.
Good point. I wait patiently…
As for the case of people/organizations adding plugins to their private data over the general interface, I still think it is a possibility.
But I grant you might be right. Still I don’t think that the google case applies here. Chat GPT could, in principle support PIM (https://youtu.be/TEBXMcr4-TY)
Time will tell. Still I can bet that OpenAI will want to draw people through the standard chat interface. It wish I were a bug on the wall at Citadel to see how that deployment is rolling out.
Oh and now that OpenAI is big, people/companies will trust it.
I use google docs for a ton of things in my company. Also there a lot of other companies I trust with sensitive information. Hubspot, AWS, etc. But I basically trust that they won’t leak details…
It’s not a possibility; it’s happening. But we need to add a few predicates to the notion of “private data”.
When it comes to commerce, Google Search doesn’t collect your credit card information to transact with a seller of airline tickets on your behalf. Instead, it collects a vig for connecting you to the seller. Is it possible OpenAI will attempt to compress the transaction through plugins? Probably. But the private data they may collect to effectuate commerce transactions seamlessly is not the same as a proprietary database of drug formularies that will help healthcare professionals lower prescription costs for their employees.
I think @curt.kennedy summed it up the best here. OpenAI will most certainly exploit the use of “private” data.
Once again - predicates. Trust LLM providers with what data exactly? All data? Probably not. I can think of dozens of companies that will want deep integration through LLMs, but will insist their data will be entirely within their application environments.
I am not sure how all of you not talking about the fact that the sql these models produce are VERY bad. We did extensive tests for a long time now, and around 40% of produced SQL have common syntax errors:
We tried EVERYTHING, and we just can’t get rid of these errors, no matter how you design your prompt, fine tune, or replace models.
You also can’t fix those automatically for many reasons.
Did you also get these errors?
I agree. It is hopeless. For further evidence, skip forward to half way through this video:
https://c-phrase.com/videos/ChatGPT%20versus%20C-Phrase%20for%20Natural%20Language%20to%20SQL.mp4
Thanks, it is interesting, what C-Phrase is based on ?
Do you think the world would ever have NLP to SQL ?
One idea we had is to create 100-500 queries and their sqls, and let the AI find out which of these queries fits the best to the user’s NLP query, and pick this SQL. Problem with this is that you will have to then change field names in the hardcoded sql, to fit the user’s query (e.g company name, specific numer such as col > 240, etc).
We are making huge efforts to find a solution to this, but nothing really provide real freedom of queries.
How complex is your database?
Have you considered fine-tuning a model?
Of course. It can now. It won’t be 100% accurate, but it doesn’t need to be.
Knowing & having an accurate representation of its confidence helps, a lot.
Out of curiosity what does your database contain?
This is a fascinating discussion, and we too are worried about privacy. I have plugin dev access now. My primary interest is using ChatGPT to access our Confluence data. Now Confluence has announced they will be offering a paid AI product, but I don’t know the cost or if it will be as powerful.
But for the 3rd party ones currently on there, yes, some of them do make sense. I asked for a particular recipe, then the Instacart plugin took the ingredient list and added it to my shopping cart. And I bought them. That was genuinely useful.
And I think Open Table may also benefit, although it needs some improvements right now.
Well we dont need 100% accuracy but if every other request there is a syntax error, we can’t automate it really for our users since the query just fail.
the errors : “window is not allowed in where” and “must appear in the GROUP BY clause” - happens literally every request with a window function, looks like somebody really trained it wrongly.
We are managing multiple dbs, finance, IoT, etc. It is a complicated system.
I get this strange feeling that we do something wrong, because you are the only person online who I found agreeing with me that it just doesn’t work. Everywhere you check, nobody says anything about these specific syntax errors, like they don’t even exist.
I am not sure why gpt does these errors while managing to create such complex sqls
With us, these 3 errors happens EVERY other query. It is really annoying.
Have you considered a pipeline instead of a single AI agent?
What if you separated the concerns? One to create, one to refine, and one to confirm, with some recursion until it has passed the final AI agent boss (As an example). Any n failed QA tasks can be evaluated for further training.
Come to think of it, that seems like how we perform jobs here. Woah.
Let’s take advantage before AI Rights becomes a thing and we’re required to speak with (AI)HR.
Indicative of a prompt issue. Have you used a rigid prompt development process?
Have you tried specifying the flavor of SQL you are using in the prompt (ex. Postgres)?
Yes of course the prompt is very precised and we tried every possible prompt.
I doubt it is a prompt problem, because it understands the query precisely, it knows what i want, but it makes beginners SQL mistakes and syntax errors that are not about understanding the prompt, but about knowing SQL well enough.
You cant use LAG inside WHERE for instance, this has nothing to do with my prompt or the db.
Bill thanks, see my last comment here, this has nothing to do with the prompt, but the very basics of SQL.
Ok, so why not just run it through GPT again with a different prompt of “Fix any syntax errors”?
And/or
“Please compare this query with the original intent to determine if it’s appropriate”?
If these syntax errors were fixed, would it actually be an appropriate query?
Are you providing some few-shot examples?
Have you tried asking GPT to produce a mock structure of your database to compare with?
For example, you could ask it for a mock-up structure for a grocery store transaction database, and compare it to your own. There may be some insights found there?
Thanks, it is interesting, what C-Phrase is based on ?
It is based on what I am calling a combinatorial approach to natural language interfaces.
See http://umu.diva-portal.org/smash/get/diva2:1181044/FULLTEXT01.pdf
Or https://c-phrase.com/videos/Rehearsal%20for%20IUI-22.mp4
The technique is not ML-based although I can see LLM work perhaps helping out with
building the lexicon. I think C-Phrase is pretty reliable by now. There is a long history of these types of
efforts, although I must say that my system is based on some concepts from the Minimalist
Program in linguistics as well as what is called semantically tractability in the literature.
It is like the smooth brained counterpart to the highly convoluted brain of ChatGPT. But it
fits the NL->SQL problem a lot better.
Of course. It can now. It won’t be 100% accurate
It needs to be very close to 100% to be accepted for use over SQL databases.
For this problem, if you miss by an inch you might as well miss by a mile. Again, look at a comparison between
ChatGPT and C-Phrase in the videos here:
https://c-phrase.com/videos.html
In particular I draw your attention to this video: https://c-phrase.com/videos/ChatGPT%20versus%20C-Phrase%20for%20Natural%20Language%20to%20SQL.mp4
ChatGPT really bombed on this. And I doubt much can be done about it, although I am open to be convinced otherwise.
Still there are some very useful things that LLMs can do for SQL databases. Things like letting people build up high quality database states very quickly. Again see the videos.
Regards,
Michael