Linking Embeddings For Large Article?!

I hope this is ok to post.

I have a WordPress site and use a plugin called AI Engine to manage my OpenAI application. This links with Pinecone to store word embeddings - everything so far is ok! My problem is that I would like to embed quite large documents for my chatbot to reference so need to chunk them. I have split one article into 9 chunks of a few hundred words each with the titles being:

Article Title - Foreword
Article Title - Recommendation 1
Article Title - Recommendation 2
Article Title - Recommendation 3

down to recommendation 8. The problem is that my chatbot doesn’t recognise that these are linked so when a user asks a question about this article I get quite a disjointed reply.

What would be the best way to create embeddings for this? Thank you for your help, spent days on this and getting nowhere!

Welcome to the forum, Craig.

I think you might find this, and indeed, the entire thread very useful. It covers the topic of overlapping to maintain context in embeddings. The length of the embedding contents - #23 by curt.kennedy

You can of course add extra sections to your embedding like meta headers and even create a short summery and include that with every prompt… if you come up with a great way to do, please share it with everyone, we’re all learning, this is all new.

Thank you for taking the time to reply. I’ll read that now and will keep you posted if I come up with a cunning solution!

1 Like