I a currently investigating building a customized learning solution that utilizes GPT 3.5 Turbo or GPT 4 api.
The scope of the project is to provide personalized learning (let’s assume a theoretical subject that is suitable for an LLM - with lots of text) at scale.
I have a very rough outline of how this could work.
• User takes a test
• System ranks his level
• Appropriate (according to user’s level) content is served
• AI chatbot is always available (answers according to user’s level)
• System can answer questions but also ask / suggest external learning resources
• After a predefined time period or on demand the user takes another assessment test
• If / when a user reaches the desired level, course is considered completed
Fine tuning the system with custom data is an option i am investigating as well.
What bothers me is how i could handle session control, conversation memory & user progress / data retention. I also want to maintain a curated list of learning resources (URLs mostly) that the system will propose to the user. I do not want to only provide text since it will be rather discouraging even for self motivated users.
Would you think that what I am proposing is doable? Can you identify the added value for the trainee? What platform would you use? Any ideas / comments will be highly appreciated.
What you are proposing is potentially doable, but it is a large project which would require regulatory approvals, a significant amount of field testing, UI/UX design, database management, back and front end code building. Education is a high growth area for AI, but also strictly controlled.
My “responsibility” is to check the technical feasibility of the project, especially regarding the “AI” part. Back / front end, DB are not of my immediate concern. Regulatory approvals are not my problem at all.
My initial hunch is to use Power Apps platform which is already available and i can reach trainees easily through it. I am not sure if i still need to use a paid connector in order to call GPT api or i can do it via Azure with no cost per trainee.
Regarding the user’s level ranking, do you think the LLM can be trusted to make an assessment? I am talking about general interest topics and i guess it has more than sufficient knowledge.
I suggest you do the test taking and assessment outside of AI (with a normal app you would have to write). There are standardized tests available to determine grade level. Once you have the grade level of the student, use AI to generate tests for that grade and have AI determine correctness, and when the student is in error, suggest what she did wrong, and websites that can help.
But the answer to your questions depends a lot on who your target user group is, are you thinking about creating something for a formal educational setting or are you thinking of this as a self supervised learning platform? (Like skillshare.com)
The target group is the whole personnel of my organization. I am not sure if we want this solution to be part of the formal education offering. Maybe it could create a certificate once a user reaches a certain level of proficiency - as it is assessed by the final exam.
I did an interesting quick test with GPT iOS app with custom instructions (which is the api system_role equivalent for “simple” users, if i am not mistaken).
These were the instructions:
Act as a soft skills instructor for the personnel of a large corporation. You need to perform the following:
Be friendly but professional. Ask the user’s name.
Provide user with a detailed (at least 15 multiple choice questions of beginner difficulty) initial assessment test. You need to establish if he is familiar with the theory and also ask him for a self assessment.
Rank his level (scale 1 to 5)
Serve appropriate (according to user’s level) content
Answer questions but also ask / suggest external learning resources
On demand provide a test to the user so that to decide about his ranking again
Let the user know about his results.
What particularly impressed me was the role playing capability. I would say it is far superior than doing it with a human instructor and of course scaling is the key word here.
For the solution we are talking about chat history must be preserved (ala chatGPT) and i have no idea if / how one can do that with the api. How can someone continue a conversation when using the api? We wouldn’t want the user to be asked for his name or to take the initial test in subsequent chat sessions.
All in all, i totally agree it is a big project that need lots of work. I am only scratching the surface at the moment.
Thanks a lot to everyone who replied and i would love to hear more opinions / insights / comments.
Thanks, could you be a bit more specific? What kind of personnel and organization are we talking about?
My best advice would be to steer of the whole “formal education & certification” thing, such a thing is indeed a very large scale project, with lots of regulatory requirements.
When it comes to assessments, GPT is much better at language focused tasks than it is at reasoning, so the quality of the “grade” provided by gpt will also reflect that.
That seems sensible, I think what you need is a kind of “company assistant” that can help everyone at the bank with information on stuff like “what’s our policies on X” and “what kind of account solutions are available”. I recommend you look into RAG (retrieval augmented generation) for this purpose.
Yep, don’t trust GPT for this task, you and your fellow employees are already much better at that, than GPT is
I understand what you mean, but my best advice is still to focus on information retrieval, because you will still need this context, or the whole teaching/learning aspects will just be pure hallucinations.
I was practicing with something very similar to your needs and outlined solution.
But different approach: For tests, AI generates the questions and 4 alternate answers and marks the right one (multichoice type). It is given in a specific format (for example json). Then i used that responses from AI to insert* into Moodle elearning platform database, so the evaluation appears in Moodle and student take the test (and be scored) there.
Something similar for the creation of content of customized learning course. The thing is that you need to know the structure of Moodle database (chatgpt could help in that) to automate the insertion of that data.
Note: iteraction with Moodle database is done with backend app (python in my case) that is also interacting with AI to get the content.
Sorry to be late replying to you, i’m in a heavy project now.
Well, my solution was a proof of concept that i was dealing with for a Moodle based service.
As i said there are 3 main components: LLM OpenAi GPT, python app and LMS (Moodle).
I considered the best approach to move to LMS the evaluation as in there the user just have to open and do the test and Moodle evaluates it automatically.
GPT 3.5 or 4 (via API from python app) is in charge of generating the questions based on its knowledge and also generate the candidate answers along with the right answer.
(Here would be the best approach to feed GPT with company own information, as someone said, so it can elaborate more customized questions of the test.)
Python app receives the gpt answers in json format (something that is asked in the prompt) then those are used to insert in Moodle database (here it is replacing the manual process of creation of a test in Moodle)
Feel free to ask whatever info you may want and I’ll try to help as the extent of my knowledge or little experience)
