Custom Made Chat GPTs for creating agency policy and administrative rules

What recommendations does anyone have for using a LLM generative AI model (a ChatGPT) to review an agency’s policies and administrative rules to ensure that they are in compliance with the governing statutes and regulations, a task of significant importance?

Would a custom ChatGPT be the best way? Does this require coding?

Is providing instructions, specifying data sets, and prompting the ChatGPT considered “training,” and does this ensure 100% accuracy in the responses?

What instructions would I provide the ChatGPT to properly train it to analyze a document I provide and determine whether its content meets the requirements in the data set, i.e., specific statutes and regulations?

Would I upload PDFs of the statutes and regulations, or can the ChatGPT find the most current versions online?

What data would I need to provide the ChatGPT?

Given that the ChatGPT only uses the data set I instruct it to use, and assuming I have verified the data’s accuracy, what are the potential limitations or risks of the ChatGPT making mistakes or hallucinating?

Welcome to the community!

What you’re asking for is a topic of ongoing debate. IMO it’s unlikely that you’ll achieve a high level of reliability with custom GPTs alone.

This is a topic looking at how to ingest long documents. This specific post concerns itself with comparing regulatory documents: Using gpt-4 API to Semantically Chunk Documents - #145 by jr.2509

If you want useable results, you’ll likely have to get your hands dirty with code, and get into the nitty gritty with embeddings and the intricacies of LLMs.

If you choose to accept this mission, you’re in the right place though :slight_smile:


Thank you. That is helpful. I need some very advanced training.


I disagree!

you just need:

  • Ten percent luck
  • Twenty percent skill, and
  • Fifteen percent concentrated power of will

I think most courses and tutorials out there are quite trash and borderline scams. The more decorated the expert, the higher the BS concentration in their talks (IME). The basics aren’t that complicated, and the rest is mostly intuition work you get by just using and evaluating the tech.

1 Like

When the GPT responds with a concordance to policy have it also state the chapter and verse of the relevant policy document(s). You then do a second reverse pass with the actual text. Works for me in checking code of conduct and other policy determinants.

1 Like

Thank you. That is really helpful.