Have you considered or did you know
- ChatGPT plugins are very very new to the world.
- That getting a prompt to use a ChatGPT plugin is not as simple and direct as a API call in regular code.
Update
Seems that as I was writing this the functionality was changing. See this related reply.
If true will turn connecting ChatGPT to API calls more into a science than a art. - That to get a prompt to use a ChatGPT plugin instead is more likely to happen in understanding the manifest field
description_for_model
Description better tailored to the model, such as token context length considerations or keyword usage for improved plugin prompting. 8,000 character max.
- That many plugin developers, myself included, need to make much better use of the 8,000 character limit. (ref) Many only put in a sentence when so much more can be added. Today myself I learned a valuable one as noted here.
- That many of the authors of plugins are still learning the art, not science, of how to craft what goes into
description_for_modelto get their plugin to be used by ChatGPT as needed. - That many plugins do not include instructions on how to use.
- That there is a Discord channel to showcase plugins that might explain how to use a plugin.
- That a list of manifest files of plugins in the store can be found here
So while you may see the problem as you doing something wrong, maybe you need to give feedback to the creators or each plugin on what you are expecting in the prompt to trigger the activation of the plugin and each learn from the knowledge and hopefully improve the plugin. 
I for one find your question very interesting and will try to use it to make my plugin better for those like you with this concern.
Side note:
With my change to the description_for_model today, this is now being generated in the ChatGPT completion on each occasion before the plugin is called.
Let’s compile this code using the SWI-Prolog plugin to ensure it’s correct.
For much more detail on the SWI-Prolog REPL plugin see this SWI-Prolog Discourse forum topic.