Hi I am fairly new to coding and LLMs (I am a 2 year CS student), and I have developed an android app that is meant to answer questions based on a manual. The manual is 556 pages long and I understand that might impact performance.
The manual has Codes with explanations. When given the codes and asked to produce explanations linked to them, the 4oMini does fairly well. However conversely, when given a situation where the LLM is required to find the code and explanation which “solves” the problem input by the user, it struggles in giving the correct answer.
I am just looking to learn if there are possible ways to improve what I’ve done so far. Currently I am using the assistants API which has specific instructions to search the manual to give the answer (along with some formatting and guidelines on how to search the manual). I am looking to improve the LLM’s ability to receive a situation from the user, and give better responses. Currently it still requires a fair amount of “guidance” and keywords in the user’s prompts for it to give an accurate response.
There’s a couple of points here. First, you need to develop a better understanding what’s causing the issue. Are the responses not good enough because the wrong content was retrieved or because the model did a poor job in generating the answer on the basis of the content provided.
Also bear in mind that gpt-4o-mini can only handle so much complexity. You might to test if a different model such as gpt-4o or gpt-4-turbo returns better responses.
For my use case, most instances were not wrong, however there was a better option to pick to fit the context given. The model may lack some context into which option might be more suited for given scenarios. I am guessing this likely has to be solved with finetuning? Due to confidentiality i can’t reveal the entirety of the context but i can illustrate with an example below:
For example,
User: “I have a screw I need to secure what should I do?”
AI: “Use a hammer”
User: “Screwdriver”
AI: “A screw driver is used for screwing screws.”
User: “I need something to screw this screw”
AI: “Use a screwdriver. A screwdriver is meant for screwing screws.”
When the user does not give it full context, or “keywords” it tends to drop the response accuracy by a lot. My goal is to make this ChatBot, usable by people who are not familiar with the context.
Right. So one option in this case would be to instruct the Assistant to ask clarifying question to the user so it can obtain additional context, which in turn would improve the information retrieval. You can even consider adding a few examples in your instructions to deal with specific edge cases like in your example.
Fine-tuning would only be useful to a degree here. It could help to get the Assistant respond in a certain way to the user and to get it to more consistently ask clarifying questions such that sufficient context is supplied as a basis for the information retrieval.
Yeah I was building onto the instructions, it helped a lot. Was figuring out another way if it were possible. But i think with the current state that would be the easiest way. To input templates in perhaps another excel sheet or the instructions. But thanks for your insight man, I am really new to this and also working based off the available resources. Getting Assistants API to work on Kotlin was a nightmare in itself.