Seeking Advice on Developing a Low-Budget University Chatbot

Community
1

Hello everyone,

TLDR: Seeking advice on developing a low-budget university chatbot using a locally hosted LLM.

I’m part of a project group at university tasked with developing a prototype chatbot designed to provide students with information, about the university. As students, our aim is to build this using open-source, cost-effective resources.

Our current setup includes Ollama for hosting our local LLM, ChromaDB for the vector database, and AnythingLLM housed in Docker for seamless integration into a website.

My current setup:

  • CPU: Intel Core i7-12700K
  • GPU: ASUS TUF GeForce RTX 3080Ti 12GB OC
  • RAM: Kingston FURY DIMM DDR5-4800 Kit KF548C38-16 (4x16GB)

Additionally, at our university, we have access to a Mac Studio:

  • M1 Max
  • 10 CPU cores
  • 24 GPU cores
  • 32GB RAM

As our Language Model needs to support German, our options are somewhat limited. Preferred options include:

  • em_german_leo_mistral.Q5_K_M.gguf
  • em_german_13b_leo_Q4_K_M.gguf
  • em_german_13b_leo_Q8_0.gguf
    also we consider any variant of the SauerkrautLM Language Model
    or any other german LLM

Any advice or experiences you could share would be immensely valuable!

Thank you!

2

Sounds like you’ve made lots of design choices already. Are you using Python and Langchain as well? And I’d be interested to know what your hardware setup is (memory, graphics card, CPU), and which LLM models you’ve determined you can run on that.

1 Like
3

These initial choices are not set in stone; they were just ideas for testing.
We are open to use Python, our WebCrawler is written in Python, and we have some basic knowledge.

While I haven’t looked into LangChain yet, during my research, I came across some tutorials that utilized it and i wanted to look further into it.

My current setup:

  • CPU: Intel Core i7-12700K
  • GPU: ASUS TUF GeForce RTX 3080Ti 12GB OC
  • RAM: Kingston FURY DIMM DDR5-4800 Kit KF548C38-16 (4x16GB)

Additionally, at our university, we have access to a Mac Studio:

  • M1 Max
  • 10 CPU cores
  • 24 GPU cores
  • 32GB RAM

As our Language Model needs to support German, our options are somewhat limited. Preferred options include:

  • em_german_leo_mistral.Q5_K_M.gguf
  • em_german_13b_leo_Q4_K_M.gguf
  • em_german_13b_leo_Q8_0.gguf
    Alternatively, we can consider any variant of the SauerkrautLM Language Model.
4

Microsoft last week released its Phi-3 small language model, which is an open source model. I’ve only tested it briefly via a chat interface but it does indeed seem to have potential and it supports German (I just tested it). Inserting a few links here in case this sounds interesting to you.

3 Likes
5

Thanks for the info. My 2 cents worth of knowledge would be to definitely go with Python and Langchain if you can. I think that’s the most common language and library, where all the work is being done. It will let you connect to not only your local models, but Cloud AIs too, if you ever want to. Langchain is the leader. And as far as open source models, HuggingFace will be your central hub to finding them and the community of open source users. Good luck. Most of my experience is with Cloud AIs like OpenAI, Perplexity, etc. or else I’d be more help.

1 Like
6

Impressive setup for your university chatbot project! Using open-source resources and hosting a locally LLM sounds like a cost-effective approach. Have you encountered any challenges integrating the various components so far? Excited to see how your chatbot prototype will benefit students at the university. Good luck with the development process!

7

Hello! I’ve been working on a similar project with overlapping goals.

I’m also a university student working on a chatbot project on my own. So far, I’ve managed to develop a retrieval-based chatbot that can answer mostly correctly based on retrieved data. Right now, I’m focusing on enhancing this chatbot’s capability to be aware of past interactions. Since I’m not an AI expert and have had to self-teach everything, development is quite time-consuming for me. Therefore, I would greatly appreciate exchanging knowledge here.

I’ve been using em_german_leo_mistral.Q5_K_M.gguf for some time and I’m personally satisfied with its understanding and responses. Due to time constraints, I haven’t had the opportunity to test many German-exclusive models.

8

For reliable data retrieval we created a CSV file using Excel to document the most common topics and FAQs for the Vector database. Additionally, we made separate text files for the faculty and staff at the university.

For our language model, we chose the standard LLama3 (8b).

Our project is now done and in preparation for our presentation, I tested all the questions beforehand to ensure the reliability of the answers.

We decided against allowing the chatbot to learn from past interactions to prevent users from manipulating it. We also tested the chatbot’s performance using only data from our vector database versus using both the vector database and the LLM’s inherent knowledge. Each approach had its own pros and cons.

I am not sure what software and framework you’re using. Our setup was adequate for a prototype, but it likely wouldn’t be sufficient for actual deployment. The IT department at our university will continue the chatbot project and plans to use the RASA framework.

Since you are also German, we could connect via Discord or another platform, and I can provide further details about our project.

1 Like
9

Thanks for your reply :slight_smile: Please excuse my late response as I have been on holiday.

Connecting via Discord sounds great! If you are still up for it, please add me on Discord and we can decide on a time and date. My username is “miaumiau1255”.

I look forward to hearing from you!