How to the API to extract common questions from a long list of conversations

API
1

I want to use an LLM to extract common questions from a long list of conversations that we had with our clients. Any ideas on how can I achieve this?

2

The best way to go about this would be using embeddings and a vector database.

You can embed the common questions using the open AI embeddings and then match the conversations against those to find the most similar ones and then extract those statements

3

Gi udm17!
Thank you for your reply.
Well, the thing is that I don’t have the common questions/phrases. I need to get them from the texts. ChatGPT is working very well manually, but I need to automate this somehow. Split the texts into chunks, query the LLM for each chunk, and then query again the results from all the chunks.

4

Hi, I would

1 split the conversations into chunks,
2 run a prompt on each chunk to extract all questions from the clients present in the chunk
3 embed the extracted questions and put them into a sort of vector db
4 clusterize all questions in db using the required distance (how far the questions may be apart to stay in the same cluster)
5 get each cluster out of the database with all questions in it and run a prompt to find the common base of all questions in the cluster (what’s the price? How much does it cost? What if buy 3 of those do you have a special price? = The price and discounts for bulk orders)

Then you have a list of your common questions to work with

5

Hey Zadrian, happy to help and walk through a solution. Feel free to DM me.