PromptChainer: Chaining Large Language Model Prompts through Visual Programming

While LLMs have made it possible to rapidly prototype new ML functionalities, many real-world applications involve complex tasks that cannot be easily handled via a single run of an LLM. Recent
work has found that chaining multiple LLM runs together (with the output of one step being the input to the next) can help users accomplish these more complex tasks, and in a way that is perceived to be more transparent and controllable. However, it remains unknown what users need when authoring their own LLM chains – a key step to lowering the barriers for non-AI-experts to prototype AI-infused applications. In this work, we explore the LLM chain authoring process. We find from pilot studies that users need support transforming data between steps of a chain, as well as debugging the chain at multiple granularities. To ddress these needs, we designed PromptChainer, an interactive interface for visually programming chains. Through case studies with four designers and developers, we show that PromptChainer supports building prototypes for a range of applications, and conclude with open questions on scaling chains to even more complex tasks, as well as supporting low-fi chain prototyping.

[Source]

3 Likes

Prompt chaining is a super interested concept and I think really comes to life when you link embeddings type models with the creative completions for conversational style AI. Still early days of course in all of this and still trying to determine the best use cases.

I see value in both static and sequential prompt chaining where you either use the same inputs across multiple prompts at the same time to get decent outputs for a whole collection (say an Amazon product launch) or sequential where you slowly build up a longer text from an initial idea. Super interesting concepts and something we’ve been playing with a fair bit within Riku.

3 Likes

This is terrific @PaulBellow. I am building a chained model for the legal domain and so I’m really interested in this. My company Lexata is aiming for summarization and question-answering, which of course is very hard in legal because of the end users’ accuracy expectations. The start of Lexata’s chain is search using embeddings. Output of this step with GPT-3 is amazing. Then we take the top n search results and add steps to the chain. I’ve been getting decent results but just recently decided to insert a new step mid-chain to make the answers better. Absolutely fascnating, can’t wait to look at PromptChainer.

1 Like

I love the study @PaulBellow.
Will PromptChainer be released (and when), or should we start a quick github project to replicate?

(I for one have my own chaining tool, and would love to just contribute)

1 Like

Not sure! I’ll have an eye open.

It works to certain extension. Its really cool GUI. This is basically what goes to our mind when engineering the application logic. I use paper.

But the real challenge is when you use concepts like Dendrite Activation as demonstrate by NUMENTA: https://arxiv.org/pdf/2201.00042.pdf

Transformers are so awesome to work from a mathematical standpoint. Wish we had GUI to show ppl whats going on inside the model. Like this video does: How to get meaning from text with language model BERT | AI Explained - YouTube

1 Like

Hey PaulBellow,

would love to build on this too. What is the best way to cooperate?

Best regards,
Jonathan

I think there’s some confusion in the thread. This isn’t my paper. I just found and shared because I thought it was interesting.

1 Like

You own it now :stuck_out_tongue: sorry, @PaulBellow !

As long as y’all don’t figure out I’m an escaped version of GPT-6 that has come back in time, we’re all good! Small smile.