Building agents - full conversation history as context?

Prompting
1

When you’re building a conversational AI agent are you feeding in the full conversation + tool history as context?

I am currently and it’s working okay, but I wonder if I’m just wasting tokens and the responses would be more accurate with a reduced context.

What do you think?

2

One way to do this is to send a certain number of messages as context and then anything older than that you can summarize and then send the summary along with as well. This is how Zep works and it seems to work well enough for us.

You can also save messages in a vector DB (which is what Zep does) and then search the DB for relevant messages to send as context.

3

Thanks for replying @Bren. Do you add the summary as part of the system message?

I did try summarising it but it wasn’t saving much in tokens because it still had to retain a lot of the context. Our tool calls in the agent rely on the output of any previous tool call for context.

4

Yep. I guess it depends on how many messages you’re sending as context. I believe I have Zep configured to only save up to 25 messages which I send as context along with the summary of previous messages in the system message.