GPT Automatic Task Planning

Prompting
1

Hey All,

I was doing some research into how the some of these frameworks create tasks for the AI to follow. One of the better frameworks I have used is Semantic Kernel, which is opensource, so I figured I’d dive into the code and check it out.

I suspected they were just using AI to create the tasking for itself via prompt templating, and it seems that is exactly what they are doing. Here is the code responsible the task creation. If you take a look at line 10 you will see their prompt.

It is pretty basic, but very effective. I’m thinking there are probably more powerful methods to create really nice task creation. But that is what I will be doing for the next 2 weeks or so in my spare time between work.

I wanted to put it out there just in case anyone was wondering the same thing, but also a semi “call for ideas”. If anyone has any thoughts or papers trying out different things, I’d definitely like to take a look at them. Specifically, if you have any thoughts into how the OpenAI is doing the plugins creation, I’d definitely like to hear your thoughts. I have a feeling they are chopping up the OpenAPI and smooshing it together with the descriptions and function names. When I was looking at the debug console in the plugins development, my function names were reformatted a little bit, so I think the plugin interface description undergoes some sort of descriptive transformation. But it is just guesses at this point.

1 Like
2

Nice. One thing I’m curious about is plan evaluation and failure recovery.
I’ve notice, wrt plugins, chatGPT will sometimes try a plugin two or three times, then try a different plugin with overlapping functionality (e.g. ScholarAI then web search).
Seems to me like two alternatives:

  1. generate plans including flow control (loops, conditionals) up front - doesn’t look like Semantic kernel’s prompt can do that.

  2. have some evaluation / recovery mechanism post-execution. Can that be part of the plan itself, or a separate parallel process? What would it look like? Can the LLM generate that too?

2 Likes
3

That’s actually a good point. That single shot plan generation is pretty effective but now that I think about it, you are right. If the bot has enough tools at it’s disposal, maybe conditional plans with built in machine logic instead of inferred logic, it could open up the possibilities while reducing costs and increasing stability.