Use embeddings to retrieve relevant context for AI assistant

[sps]

@thinktank

What sort of files can you store your conversations in?

Currently this code writes to the specified file as csv, but you can alter the code to go with JSON or any other format that works for you.

@thinktank

Is there some way to intelligently structure those files so that ChatGPT can more efficiently look up the information later?

For lookups at scale, you can use vector databases
I’m using csv for the fact that it allows me to append new conversations to the file without having to read the file, which same time and compute, and of course the code I wrote can be further optimized for speed.

@thinktank

So, let’s say you have a long conversation on a given topic that would exceed your token limit. You’d summarize the output then save it. Does an embedding with find_context() know how to return to files it’s already made? Or can we help it along by providing more structure in the summaries provide

This code doesn’t summarize the conversation, it just stores every message along with embeddings and retrieved the most relevant ones as specified by the mask you can write own criteria for choosing the conversation.
You can add more code to it. e.g If the relevant messages were to consume too much tokens, they can be summarized before being sent for completion.
IMO this is not the only or the best approach, it’s a demo for learners to be acquainted with embedding for conversational context.

@thinktank

Don’t embeddings result in biases in data retrieval? Is there a way to correct for these biases?

I’m not sure what you mean by bias in this context. As @anon22939549 shared, embeddings are used to filter out semantically relevant info from the rest of content.

@thinktank

In your code you’re calling on gpt-3.5-turbo. Will any part of this code change significantly when you start using 4.0? Or does this just change the nature of the instructions you pass it? I understand, GPT4 understands conversation far better than 3.5. Which I’ve seen in action, and I think is responsible for a lot of the word-count questions getting asked in Prompting.

The code is written for Chat Completions API, which means it can consume any model as of now that can be accessed over the chat completions API. You can simply change the name of the model in the API call to the chat model you want to consume.

@thinktank

I think I read somewhere that 3.5 has trouble with the “system message,” so will GPT4’s greater conversational capacity change how you can interact with it here? As in, only retrieving relevant summaries until it needs more information to try to keep within the token limit?

You aren’t wrong here, but that was for gpt-3.5-turbo-0301 the latest 3.5 model released on 0613 is better at understanding context and following system message.
In fact,

@sps

The new gpt-3.5-turbo and its 16k version is way better than the 0301 version. I have another project where I had to use gpt-4 because of the complexity, and the new gpt-3.5-turbo model excels in the same task.

However you can use gpt-4 if you still want to, no changes required in the code, except the model name.