Hey everybody!
I’m working on building a chatbot for a local bank, well a demo version. All the bot should do is act as a customer service agent that answers queries, faqs and explains banking processes plus procedures. Nothing more, nothing less.
I took two approaches; first, I built one bot that works by using fine tuned models (I have a few of them for different types of knowledge) through the chat completions api; and the other bot uses a vector store with the assistants api.
What I want to know is:
- Should I implement both approaches?
- If I am to use both, can i run my ft models through the assistant or do I run them on the assistant’s response to the user query before printing the final response?
Please explain the reasoning for your answers.
Thanks!
1 Like
Welcome to the Forum!
Basically you don’t need a fine-tuned model for this purpose and should use RAG instead. The fine-tuning process is not designed to support knowledge retention and rather serves to get the model to respond in a certain format or style.
For additional background you may also want to read the information under this link: https://platform.openai.com/docs/guides/fine-tuning/when-to-use-fine-tuning
4 Likes
Not to discount @jr.2509 but in a financial institution you may need to balance more towards accuracy if even more limited in scope… perhaps a fine-tuning approach. Airlines have gotten in trouble using chatbot approaches that were able to paint between the lines and were held liable for things like quoting errant pricing.
Just say’n, this may be a more involved question due to financial regulations.
Either way, it’s great that you built both directions. Perhaps let the client test and sign off on the chosen direction so you’re not held liable.
3 Likes
Hi @marko-1 - thanks for chiming in here. You are absolutely right that there’s more stringent requirements for the adoption of AI in financial services (I work in the field myself).
The main challenge is that fine-tuning - while useful for a lot of things - will not help with the knowledge aspect of it. If you were to fine-tune a model based on question/answer pairs, you end up having a higher risk of hallucinations then when implementing a well-designed RAG process.
That said, sometimes in order to access knowledge, function calling can and should be used. To improve the accuracy of function calling you can fine-tune a model. But you would still need to retrieve the information from an external database.
3 Likes
I honestly don’t see how you can 100% eliminate an LLM saying something off-script.
I suspect the best approach here is prominent disclaimers, possibly even one a customer has to confirm they’ve read before continuing to interact.
the more popular this style of bot becomes, the more people will be used to the odd dropped ball too …
3 Likes
This is the most reasonable response. It is also worth noting that the more people become aware of the ways to trick current LLM chatbots the more they will look for the opportunity. And as you say, airlines have already had issues… imagine if someone tricked a bank bot into saying that it will credit their account for $100000 - will that go to court, will the bank be on the hook? And if so, who in the organization gets the blame?
From what I see on this forum and other places, it seems like many people don’t understand the downsides and pitfalls of this new (and hallucinating) tech. For something like a bank there should be a massive amount of research into the platform and vetting by a legal department before rolling out something like a LLM chatbot for customers.
I sincerely worry for both the end consumer and the companies pushing their tech teams into deploying LLM bots in some sort of convoluted effort to save money or whatever they think is the upside of the integration is. As it stands right now this “AI” technology is not ready for prime time.
1 Like
Thanks everyone for the replies.
My takeaway is that, I should let the bank make the choice and whatever they choose I should implement disclaimers to the end users so I’m not liable for whatever shortcomings either approach has.
Note that, given what I want the bankbot to do, as I described in the initial post, using a well defined RAG process seems to be the better option, at least initially. Overtime using a hybrid approach will likely become necessary.
If you have anymore thoughts please share.
Good to know you work in that space @jr.2509
Please explain, in general, what these requirements are especially in regards to what I want my bankbot to do.
1 Like
There are currently no global, uniform requirements for the adoption of AI in financial services and you will have to consider the specific regulatory requirements in the country the bank operates in. Many countries and the specific regulators in the financial services sector are still in a nascent stage in defining AI requirements. Additionally, these are frequently not an individual application type level but rather encompass general principles that should be observed when implementing AI systems in financial institutions. Your role as the technical developer is only part of the equation - there are many other things that banks typically need to consider when adopting AI 
My recommendation would be that you seek some guidance from the bank’s compliance department regarding existing regulations or anything that’s on the horizon and that you should take into account when building the chatbot.
To give you a flavour what some of the issues are that have attracted financial regulators’ attention in relation to chatbots specifically, you might want to read through the following report from one of the key U.S. regulators: https://www.consumerfinance.gov/data-research/research-reports/chatbots-in-consumer-finance/chatbots-in-consumer-finance/
Depending on what country the bank is located in, I might have some other links off the shelf (no promises). In any case I could not give you any binding advice and the bank’s compliance department is best positioned to offer guidance on this.
3 Likes
Thanks for that information. Now I see why you recommended RAG.
1 Like
Jen is very smart on this topic… I’m learning a lot!
One additional consideration, while using RAG, is to test changes of Temperature and Top-P.
2 Likes