Plugin injection attack, pseudo code prompts, chain of thought, plugin orchestration, and more

[qrdl]

Apologies if folks are already doing this, links would be appreciated if so.

This injection attack implies that some kind of chain of thought can be done with the plugins

Johann Rehberger

@bruce.dambrosio came up with this cool idea

Prompts as psuedo-code - where are the limits?

Thought. GPT can read and write code. So why not think of prompts as psuedo-code? ie, use control-flow like for, while, untill, etc. Also how about assignment? I’ve tested simple for each loops and they seem to work just fine… Does this give us any capability we didn’t have before? I’ve never really thought of prompts as including control flow (or assignment).

Would it be possible to write sort of a ad hoc NL plugin language that can orchestrate plugins in concert?

might be too undeterministic to make it work that well for most stuff, but for certain conceptual / brainstorming tasks that could be OK

I googled and looked at the plugin docs but didn’t see anything here. plus nothing is popping out in this forum. again, apologies if i missed something obvious

[PriNova]

What you can do, and this is what I do, is to give instructions as a response from your own coded endpoint of a local plugin.
This instructions work same as a prompt. So yes, you can inject prompts into it and orchestrate also other plugins with it.

[qrdl]

can you do something like

result1 = doplugin(a)
result2 = doplugin(b)
result3 = doplugin(c, result1, result2)
print(result3)

as a prompt?

I got this to work:

Going to see if I can get GPT4 and wolfram to prove a math theorem

[PriNova]

It only works for the registered plugins. I use natural language as prompt. One response of my endpoint is:

“If plugin is not found, then retrieve all plugins from the /list endpoint and look for a similar name.
If there is a plugin with a similar name, then request all endpoints again.
If there is no plugin with a similar name, then say 'Plugin is not found!”

This is part of a plugin retriever, which has access to all 100+ registered plugins in ChatGPT with their respective endpoints.

But you could also exchange this prompt with something like:
“If plugin is not found, then use the plugins in the following order: WebPilot, Metaphor, KeyMate.AI.
If request succeeded than, answer with the response from the /executePost endpoint.”

[PriNova]

If this task is as short as provided in your example, it is fine to use pseudo code.

For longer tasks or complex instructions, I recommend using natural language, because the tokens used for code is much more than expressed in a well defined prompt in natural language.
There was someone on the discord invented Macromania as a functional prompt language. Compared to NLP it suffered a lot by eating up tokens.

And you now, at the end of the day, token is our ‘new’ gold.

[qrdl]

yeah, it’s an interesting question. We’ve been speculating that GPT4 gets its reasoning capabilities from training on so much code, so I’ve just been thinking that maybe we should talk to it in the language it’s thinking in.

[PriNova]

Not only code. Also from the text corpus it was trained on.

Deep Reasoning relies significantly on logic, but also on Information and Contextual Understanding, Critical Thinking, Cognitive Abilities, Open-mindedness, Communication and Dialogue. The most lacks in LLM.
The deep reasoning (if present) is simply because of statistical patterns and associations, but lacks true comprehension or deeper understanding. They excel at pattern recognition, but their knowledge is shallow, as they lack genuine comprehension or a conceptual understanding of the world.

So at first sight, it seems like deep reasoning a human can do, but it isn’t.

[qrdl]

This seems to work. I’m very curious how far I can push this.

I wonder if a problem might be is all the RLHF is getting in the way. It’d be cool if there was some kind of NDA protected non rlhf version we could get access to, fully recorded of course for inspection by one and all.

Heh, there probably is.

works as well:

Please follow this algorithm and don't output additional text other than what's specified below:
result1 = recent research papers about large language models
result2 = recent research papers about protein sequencing
def get_theme(rp: research paper):
    return one or two phrases that best describe rp
def print_themes(result: research papers):
   themes = []
   for each rp in result 
      themes.append(get_theme(rp))
   print(themes)
print_themes(result1)
print_themes(result2)

If we can get the above to work well, the immediate and validated benefit would be reuse, I think. Superior reasoning is trickier to validate.

[Apekool]

Up to recently it was possible to write prompts on GitHub and then import them in another prompt, e.g. importing five prompts inside of one prompt without being limited to max. characters etc. Stopped working a few days ago / gives click issues now. Not sure if a bug/issue or block.

[qrdl]

some wild and crazy prompt injection issues there for sure, but yeah, that could get fun.

One problem is that larger context windows seem to degrade reasoning, or maybe it’s just too many concepts to pay attention to at once. I am crossing fingers that chatting with algos might alleviate that a bit.

[PriNova]

@qrdl

Please follow this algorithm and don't output additional text other than what's specified below:
result1 = recent research papers about large language models
result2 = recent research papers about protein sequencing

I rephrased your code command into a strict prompt like this:

Please follow this instructions and only output the result without further explaining. Use ScholarAI.
Search recent research papers about large language models.
Search recent research papers about protein sequencing.

And the result is:

Here are the recent research papers on large language models:

1. “Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models”
2. “Paraphrasing with Large Language Models”
3. “Geant4—a simulation toolkit”
4. “How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment”
5. “Enriching Word Vectors with Subword Information”

And here are the recent research papers on protein sequencing:

1. “Protein sequencing by tandem mass spectrometry.”
2. “Automated de novo protein sequencing of monoclonal antibodies”
3. “Graphene Nanopores for Protein Sequencing”
4. “Shotgun Protein Sequencing”
5. “The emerging landscape of single-molecule protein sequencing technologies”

Your prompt with code used 135 Tokens.
Mine used 36 Tokens.

So, think a second time about if it is worth the work ( especially by utilizing the API.)
I don’t want to discourage you, like I’d said above “Tokens are the new currency.”

[Apekool]

It may cost you less tokens (when using more similar prompts) if you ‘predefine’ in a first prompt all your configurations. For example, by using one prompt to define “search QUERY” will allow you to use “search QUERY” in the next prompts by just using “search QUERY” instead of including the whole config per prompt…

e.g. you can use one large prompt to set up / configure a search and analysis terminal

And then just use very shorts prompts after that to do whatever searching and analysis you want

EDIT: did not test this on API yet but I suppose it works the same? It requires some real fine-tuning though. The web browsing plugin works a bit crappy but with trial and error it’s getting there.

EDIT2: is even possible to run multiple predefined commands with an extensive config per command in single prompt if config is set up right (can be very tiring and difficult to get the whole config right depending on what you try to do, but the result can be really nice).

{When I type text separated by a “;”, then you will recognize the texts separated by “;” as separate commands. For example, if I type “search Amsterdam ; analyze 1 ; report”, you will execute the search command on QUERY “Amsterdam”, after which you will execute the analyze command on the search result with [Data ID] = 1, after which you will execute the report command.}

[qrdl]

PriNova, you’re probably thinking in terms of productizing, I’m thinking more jupyter notebook jazz.

The potential for re-use is attractive to me, imagine importing some functions (as mentioned above) which can do some sort of interesting reasoning based on ideas that would be useful for whatever I’m looking at that moment. Composition / Dependency injection / all that fun stuff could potentially be used.

It’s also possible we could perform superior inference by specifying our logic algorithmically rather than via natural language. Yet to be proven of course.

[lachie1]

What an interesting thread! I can see some kind of prompt language being developed in the short term to save token usage.

[PriNova]

Yes, it is.
Especially when it comes to repetitive tasks within a session like the approach @Apekool mentioned. I used the same approach for Software Design and Implementing. I went some steps further and pre-prompted some kind of ‘Menu’ to choose and linked the commands in a pipe-like way.

For example if I implemented some kind of code my command chain after copy/pasting the code was: code_metric : refactor : chunking : code_metric.

This 4 step approach made it very effective working in ChatGPT.

Here was the Menu in the first version I used:

ADD Menu options:

• “next”: You always resume from the exact character you left off on.
• “refactor [function, code snippet]”: You refactor the code or function into cohesive form.
• “summary”: You provide a brief summary of all the informations.
• “symbolTable”: You provide a complete and compact list of the symbol table with return types and parameters used in this code or class formatted as a tree view ordered by class.
• “todo”: You provide a categorized list of what is already implemented and what has to be done.
• “codeMetrics”: Provide a brief metrics of the code like Complexity, Readability, Debuggability etc.
• “chunk”: Refactor legacy code into smaller modular functions.
• “menu”: You show the menu.
• “library”: A comma-seperated list of context-relevant keywords.
• “exit”: We exit the build process.

Now I’m using GitHub Copilot Chat (even for non programming tasks. Only prefix “Software” in your prompt and it could also write stories in a text file)

[bruce.dambrosio]

So, does this mean this kind of prompt can cause the model to do MUCH more work internally for the same number of tokens? Sounds like it might break openAIs pricing model of using tokens == cost. As far as I can tell from my experiments in playground, it works. For example, as far as I can tell, this ‘prompt’ goes through quite a few iterations before producing output:
(apologies to @PriNova for the sloppy psuedo-code, I’m lazy)

Respond ONLY with the final design.
designer is a function that accepts a single argument, ‘task’ and returns a single object ‘design’, a design for a system to perform that task.
critic is a function that accepts two arguments, ‘task’ and ‘design’, and returns a single object ‘critique’, a critique of the design with respect to the task definition.

queryGeneration is a function that accepts three arguments, ‘task’, ‘design’ and ‘critique’ and generates a return value, ‘query’, a question to ask the user to resolve ambiguities or priorities in design development. Question can return None if there are no ambiguities to resolve.

ask is a function that accepts a single argument, ‘query’, and returns ‘response’, the user response to query. Ask presents the query to the user, and then STOP to permit the user to respond.

editor is a function that takes four arguments, ‘task’, ‘design’, and ‘critique’ and ‘response’ and returns a revised ‘task’ with additional information intended to improve the performance of the design function with respect to the critique.

Given the task: ‘design an automated assistant for writing technical papers’,
loop over designer, critic, queryGenerator, ask, editor until the design stabilizes.

Respond ONLY with the final design generated in yaml

[bruce.dambrosio]

btw, I do notice that playground refuses to run the prompt twice without my modifying it a bit. Not quite sure why. maybe a DOS attack supression filter or something.

[N2U]

Small note on the initial Twitter post:

Why no human in the loop?

He was the human in the loop, he enabled those plugins, he could have pressed this button any time to stop it:

Not mentioning this seems disingenuous…

I personally think that being able to store instructions in html on the web seems very useful

[qrdl]

Fwiw, I’ve been known to tab away to another task after starting a query on chat. Those twitter feeds won’t read themselves, ya know.

[PriNova]

Yes and no.
It does not really operate internally. They are remembered from the context window.
And yes, you can save tokens for repeating tasks. But it forgets the function tasks as the context window slides further down the conversation. So hinting at not forgetting is sometimes helpful.

What I did while programming is to role-play Assistant into the persona of Robert C. Martin aka Uncle Bob. Assistant knows a lot about this character’s skills. That saves tokens too.

For example, during a code iteration, my question was, “How would Uncle Bob implement this peace of code? Let’s work it out step by step”.
Another funny convo I had was asking to refactor a piece of code while the assistant played the role of Uncle Bob. My question was: “Hey Bob, how would Martin Fowler do a refactor of this code?” Assistant’s (Uncle Bob’s) response was, “You do not need to ask Martin Fowler to do that. My expertise and skills are sufficient to do the job. I would…”

One more small comment on your last paragraph.
You instructed Assistant to “Respond ONLY with the final design generated in yaml”. If you give Assistant the possibility to execute your tasks step-by-step, the results will usually be better. There is a phrase that has performed very well in benchmarks: “Let us work this out step-by-step to ensure the right answer”.