Checking for what isn't; The nothing that should be

Prompting
1

I’m about to start working on a new project. The idea is to take a manual, say like a chemistry book, then query a user on their knowledge. I’m converting the documents to a vector database and then loading the conversation with the information when the user prompts the bot. Pretty standard so far.

The problem is, in my case I need to know if the user has skipped a step or missed something important. The instructions may be spread out across multiple chunks, and they might not all be present in the user’s prompt or retrieved chunks. If it were just one thing I was testing, it would be easy enough to just bake a step checker into the prompt, but this needs to be more generic. The AI needs to know based on the data and the prompt alone.

My immediate solution is a knowledge graph of the chunks and how they relate. I’m thinking that will get good results. I can keep pulling in data until it looks like it is the end of the line. But, it’s experimental and I wanted to see if anyone had any solutions that I should check out first.

Thanks

2

This might be something where your example of a chemistry book is something that can already be proctored by only AI knowledge. I assume this is new knowledge for the AI?

I suppose the challenge I see is in whether the task retrieved actually requires steps, and if the documentation of a process would require them. That the embeddings stays with the AI for the course of quizzing, or that it has another knowledge document window that can be accessed or even polled to retrieve data (maybe functions to directly call sections of the document found by embeddings TOC overview, iterating to the next part).

The AI might need to do its own thinking there to administer, just as my instruction told it to plan the speed of the quiz to arrive at the destination. You can have another AI make a hidden outline. A full chapter section of a calculus book or turbine maintenance manual may compromise your chat history ability even if it lays out a new skill.

I suppose the first thing is to think about your data, how you might make it into reasonable units that can be placed into AI memory - how you would paste into your messages from the source material if doing it by hand to get the function to work. Then ponder how an AI can assist in the chunking.