What recommendations does anyone have for using a LLM generative AI model (a ChatGPT) to review an agency’s policies and administrative rules to ensure that they are in compliance with the governing statutes and regulations, a task of significant importance?
Would a custom ChatGPT be the best way? Does this require coding?
Is providing instructions, specifying data sets, and prompting the ChatGPT considered “training,” and does this ensure 100% accuracy in the responses?
What instructions would I provide the ChatGPT to properly train it to analyze a document I provide and determine whether its content meets the requirements in the data set, i.e., specific statutes and regulations?
Would I upload PDFs of the statutes and regulations, or can the ChatGPT find the most current versions online?
What data would I need to provide the ChatGPT?
Given that the ChatGPT only uses the data set I instruct it to use, and assuming I have verified the data’s accuracy, what are the potential limitations or risks of the ChatGPT making mistakes or hallucinating?
If you want useable results, you’ll likely have to get your hands dirty with code, and get into the nitty gritty with embeddings and the intricacies of LLMs.
If you choose to accept this mission, you’re in the right place though
I think most courses and tutorials out there are quite trash and borderline scams. The more decorated the expert, the higher the BS concentration in their talks (IME). The basics aren’t that complicated, and the rest is mostly intuition work you get by just using and evaluating the tech.
When the GPT responds with a concordance to policy have it also state the chapter and verse of the relevant policy document(s). You then do a second reverse pass with the actual text. Works for me in checking code of conduct and other policy determinants.
