Trying to let ChatGPT use guidelines from a document

Community
,
1

Hi, apologies if this is in the wrong forum section.

I’m working at an E-learning company and we’re creating a lot of content that needs to follow certain guidelines, principles, etc.
I have a large PDF, 46 pages containing guidelines, examples, tips, and a checklist to check if exam questions and answers adhere to certain requirements.

Basically there are some 20 rules defined, each with a good and bad example.
I would like ChatGPT to test an exam question against each of those rules. Not just looking up each of those rules, but also applying them.

I have tried to construct prompts to use the PDF to score and change input, but ChatGPT never wants to use all of the rules, it seems to randomly pick some and be done with it. Even in a conversation, it simply will not use them all. Results vary wildly.

The PDF is probably not structured in a proper way for ChatGPT to use, and even though we have a large context window, it likely exceeds it.

What would be a better strategy to tackle this instead of trying to RAG with ChatGPT? I’m thinking finetuning on the materials, but that requires turning the document into something completely different for use with finetuning. That is/sounds like a colossal amount of work. are there any tools or strategies to make this easier? Or should I keep using RAG but in a different way?

2

You say “ChatGPT”, which is OpenAI’s chatbot product at chat.openai.com, but then ask other questions beyond its capabilities. Are you referring to the API?

If you want a model that isn’t optimized by having poor attention, you’d have to use the API’s gpt-4-0613. Then, placing all rules the AI must follow, in a clearly communicated structured manner, into the text of a system message as the way the model must operate, would be your best course for ensuring this knowledge is always present.

1 Like
3

I was referring to ChatGPT, as that is what the employees use. It sounds like I would need to construct a custom app that utilises the API instead. I will give that a go!

4

You can also employ ChatGPT’s custom instructions feature, but distilling the information you describe down to a 1500 character text input box sounds like a challenge.

1 Like
5

Hey, this seems interesting. Can you please explain this details. I new into prompting and facing similar issues. I am willing to learn. Can you please guide me in details how to do that in your method.

Thank you in advance.