The Problem
We are trying to build an AI chatbot that uses a mix of external functions and RAG for an e-commerce provider on their website that receives about 200,000 visitors monthly worth of traffic.
We are estimating a 60,000 people interaction with the bot, with an average of 10 messages, with a conversation length of 750 words back and forth in total. (this is just for customer service enquiries).
The biggest issue we have been struggling with is finding ways to cost mitigate, we are aiming for about a $0.06 cost per user interaction. On testing GPT-4 we submitted one document (about 1100 words), and on our first ask to tell us what the document was about, the cost was already around 8 cents on one interaction (we need 10 messages back and forth).
We then moved to GPT 3.5, and on our first message to the bot (that required retrieval) was only around 5000 tokens. On our third message it was 50k tokens!!!
We have also noticed that in the playground, that everytime we submit a request, we are resubmitting the annotation of the snippet time and time again, which means we are consuming 5000 tokens on our first question to the bot, but after only 3 short messages, we are using 50k tokens!!! So to achieve 10 messages back and forth with assistants, compounds to a ridiculous amount.
Our Cost Mitigation Thoughts
This is what we need some help with (and some validation in our thinking).
-
Primarily what we have been discussing as a team is whether we scrap the internal GPT RAG system, and build our own langchain to cost reduce, where we can Q&A with other LLM models that are far cheaper.
-
We are planning to call the external langchain via functions.
-
We are not sure whether the internal built retrieval function in assistants costs us input tokens to query the files that we are placing?
-
We are not sure whether there is therefore a cost benefit to using an external langchain/RAG?
I have attached some screenshots for you guys to see, but I guess the ultimate question is how we can keep costs down for our customer whilst still providing optimal experience.
Below is my conversation with the assistant:
User
give me a short description as to what the key point of coles code of conduct is about.
Coles Sales Assistant
The key points of the Coles Code of Conduct are outlined as follows:
** The purpose of the Code is to ensure a high level of integrity and ethical standards in all business practices, outlining how employees and contractors should behave in the workplace and conduct business at all times[1].*
Is there anything else you would like to know about the Code of Conduct?
[1]
User
what else can you tell me about the code of conduct?
Coles Sales Assistant
The Coles Code of Conduct outlines the expectations for representatives of Coles, highlighting the following key points:
1. Representatives are expected to treat everyone with whom they interact with dignity, courtesy, inclusivity, and respect.
2. They are to perform their roles to the best of their abilities and in accordance with Coles’ values and behaviors.
3. Representatives are expected to make truthful statements, promises, or commitments that they and Coles are able to meet.
4. Compliance with all Coles policies, standards, procedures, and relevant laws is required, as well as participating in any reference, probity, or other background checks as required by Coles.
5. Reporting any situations where there could be a breach of these expectations or witnessing any conduct in breach of these expectations is encouraged, as well as participating in complaint resolution processes honestly and if considered safe to do so.
6. Protecting the reputation of Coles and not engaging in disparaging or untruthful remarks about Coles, other team members, customers, competitors, contractors, or suppliers.
7. Behaving in a way that ensures the safety of oneself and others, dealing fairly and honestly with all parties, and acting in the best interests of Coles.
8. Not knowingly participating in any illegal or unethical activity, or entering into any arrangement that would conflict with the best interests of Coles[1].
Additionally, leaders at Coles are expected to lead by example, help team members understand what is expected of them, give feedback and coaching, address inappropriate workplace behavior, and promote strong governance and compliance practices within their teams[2].
Is there anything else you would like to know about the Code of Conduct?
[1][2]
User
awesome, i’d like to know who is at most risk for breaching this code of conduct
Coles Sales Assistant
[2]``[1]Based on the information in the Coles Code of Conduct, individuals who may be at most risk for breaching the code include those who engage in behaviors such as harassment, discrimination, bullying, vilification, victimization, serious safety breaches including physical or psychological violence or threats of violence, misleading or defrauding Coles, theft, misuse or wilful damage of Coles property, and serious and/or repeated breaches of Coles policies, standards, procedures, or the law.
These behaviors are considered inappropriate workplace behavior and can lead to actions such as termination of employment without notice if substantiated. Therefore, individuals who engage in these types of behaviors may be at a higher risk of breaching the Code of Conduct and facing serious consequences as a result【16†source】.
Is there anything else you would like to know?
[1][2]
And here is the screenshot of the token usage:
