Linking Embeddings For Large Article?!

Community
1

I hope this is ok to post.

I have a WordPress site and use a plugin called AI Engine to manage my OpenAI application. This links with Pinecone to store word embeddings - everything so far is ok! My problem is that I would like to embed quite large documents for my chatbot to reference so need to chunk them. I have split one article into 9 chunks of a few hundred words each with the titles being:

Article Title - Foreword
Article Title - Recommendation 1
Article Title - Recommendation 2
Article Title - Recommendation 3

down to recommendation 8. The problem is that my chatbot doesn’t recognise that these are linked so when a user asks a question about this article I get quite a disjointed reply.

What would be the best way to create embeddings for this? Thank you for your help, spent days on this and getting nowhere!

2

Welcome to the forum, Craig.

I think you might find this, and indeed, the entire thread very useful. It covers the topic of overlapping to maintain context in embeddings. The length of the embedding contents - #23 by curt.kennedy

You can of course add extra sections to your embedding like meta headers and even create a short summery and include that with every prompt… if you come up with a great way to do, please share it with everyone, we’re all learning, this is all new.

3

Thank you for taking the time to reply. I’ll read that now and will keep you posted if I come up with a cunning solution!

1 Like