It’s not a possibility; it’s happening. But we need to add a few predicates to the notion of “private data”.
When it comes to commerce, Google Search doesn’t collect your credit card information to transact with a seller of airline tickets on your behalf. Instead, it collects a vig for connecting you to the seller. Is it possible OpenAI will attempt to compress the transaction through plugins? Probably. But the private data they may collect to effectuate commerce transactions seamlessly is not the same as a proprietary database of drug formularies that will help healthcare professionals lower prescription costs for their employees.
I think @curt.kennedy summed it up the best here. OpenAI will most certainly exploit the use of “private” data.
Once again - predicates. Trust LLM providers with what data exactly? All data? Probably not. I can think of dozens of companies that will want deep integration through LLMs, but will insist their data will be entirely within their application environments.