Writing a ChatBot (not just for Q&A) is hard! 2 months in and still unsuccessful :/

I feel lost and looking for some guidance :pray:

Goal: I’m trying to build a chatbot to help small business answer clients as they used to.

The basic method of: embedding (pricing, about us, hours…) + “You are a virtual assistant” system message + “Answer the user message based on the context below…” in a user message; ** doesn’t do the trick** for me.

Input: I get all their old conversations with clients + some rules (e.g. prices).

Desired output: the bot greets people the same way the business owner does, asks them the same follow-up questions, responds the same as the owner did in different situations, and sometimes prints “leaving for the owner” in specific scenarios (like a client that wants a phone call).

Are there any best practices to approach it? Any ideas on how to approach it?

1 Like

Can you share your exact prompt and an example of the output that you’re not liking?

I’m assuming that the core issue is, the model doesn’t know anything about the business so its using its world knowledge to answer questions in the context of what it thinks are similar businesses. You have to ground the model with facts to prevent this. That means using semantic search to pull in relevant facts either scrapped in from their website or given to you via a super detailed questionnaire.


There are only two options to do what you want, at least for now.

The first is what @stevenic proposed:

  • Tokenize all your clients’ support data and store it in a vector database such as Pinecone.
  • Use similarity search for each message from the client to find relevant data in the vector database, and embed it into the prompt that should generate the response.
  • Include the history of the conversation and instruction in each prompt.

The second option involves fine-tuning. Basically, you can upload your documents/history of support requests/responses to OpenAI’s server and they will retrain a model with your data. However, this approach has some limitations as well. For more information, please refer to the documentation:


For more direct guidance on how to approach this, try langchain.


I gave it a shot and got frustrated that i couldn’t see what prompts were sent to the api. I should probably try it again.

I have built some python apps and as well as Azure solution.

Hope the YT channel below can prompt some new ideas?

What I found really works well on a custom private data is this Azure solution. It only provides the answers, but also citation to the exact page.

To see the prompts you need to turn verbose flag ON:

chain = LLMChain(llm=llm, prompt=prompt_summary, verbose=True)

Also, you can turn on the debug mode:
langchain.debug = True

This is the link to free course[LangChain for LLM Application Development] https://learn.deeplearning.ai/langchain/lesson/1/introduction

Check out “RASA GPT” a fork of the open-source RASA business chatbot: GitHub - paulpierre/RasaGPT: 💬 RasaGPT is the first headless LLM chatbot platform built on top of Rasa and Langchain. Built w/ Rasa, FastAPI, Langchain, LlamaIndex, SQLModel, pgvector, ngrok, telegram

Most of the hard work has already been done.


I am considering implementing the azure search openai demo.

  1. How accurate are the results related to the search string?
  2. How is the performance in a multi user environment?
  3. What are your monthly azure costs for this solution?
  4. Does it require deploying both a chatgpt model and a gpt3.5 model?
  5. Does it only read from PDFs, or can you have it read from a database?

Thanks in advance.