The problem is if you have the first 100000 document training you pay for the 100000 document training. Next time you make a correction with 5000 documents you must suffer 105000 documents instead of 5000.
If you’re finetuning with that much, you’re probably going overboard. Remember that finetuning takes a minimum of 200 samples (not documents). And finetuning with DAVINCI only costs $0.03 per 1000 tokens at training time. Depending on the task, each sample may only be 100 to 300 tokens, so if you multiply 300 tokens by 200 samples you get 60,000 tokens total which is $1.80. Even if you go 10x that size to 2000 samples, that’s only $18.00, which is a pretty trivial cost in the grand scheme of things.
Furthermore, if you’re starting with a sample size of 100,000, adding another 5% is probably not going to modify your performance much if at all.
I’m curious why you think you need 100,000 documents to begin with, and why you think you’d want to retrain with just 5% more samples? It’s important to remember that traditional ML pipelines don’t really apply to GPT-3.
I will now try to answer but i have to prepare you that my english is bad.
My example with 100000 is certainly exaggerated but not impossible. If you train one chatbot it is certainly not possible to train with 200 samples.
Open Ai made an example by itself with learning a new topic
This example is about 713 unique pages. In my opinion these are 713 A4 pages of text.
I personally think that GPT 3 is a great tool and I like to experiment with it. However, my experience of 4 months shows me that the statement that with 200 samples it is possible to make professional tools is as probable as the advertising statement “make a chatbot in 20 minutes”
Well, I’m sorry you have found it to be so difficult. I succeeded in making a very powerful chatbot with just over 200 samples. It was able to provide compassionate listening on a broad range of subjects and even provide life coaching. Happy to share the finetuning data with you.
I still think you’re thinking about GPT-3 like a conventional ML model that is starting from scratch. Remember that GPT-3 is already capable of out-performing humans on many tasks. All you have to do with finetuning is teach it to focus on a specific task. You don’t have to teach it from the ground up.
Update: here’s a demo of the chatbot I trained with 200 samples on CURIE:
User: Hey Eve can you help me learn to eat better?
EVE: Of course! What do you want to start with?
User: Teach me the basics of cooking and eating!
EVE: Well, cooking is about preparing food in a way that best preserves its nutrients and taste. There are lots of different ways to do this, like baking, frying, roasting, and so on. The same is true for eating - there are lots of different ways to eat that can be healthy or not. For example, if you have a diet rich in animal products, you might eat a lot of red meats and cheeses. If you have a diet rich in processed foods and starches, you might eat a lot of bread, pasta, and sugary snacks.
User: Okay that’s great but teach me the basics, where do I even get started?
EVE: Well, the first thing you want to do is to eat a lot of fruits and vegetables. They’re both great sources of nutrients and they taste great, so eat them up! Once you start to feel more comfortable in the kitchen, you can try making some simple recipes, like smoothies and baked goods.
User: Okay how do I cook vegetables? What do I need?
EVE: What kind of vegetables?
User: I dunno like carrots I guess
EVE: Well, the first thing you want to do is to wash the vegetables really well. Next, you want to cut them into slices or chunks and put them in a saucepan or skillet with some water and a little bit of salt. You can also add some olive oil, but this is not necessary.
User: Okay cool, I want to talk about something else. My sister has been stealing my xbox. How do I handle that?
EVE: Well, it’s important to be understanding and compassionate, but you also need to be firm. Try to understand what is causing the problem and what your sister needs, but keep in mind that the most important thing is to keep your xbox safe. You could try talking to her about it, or maybe even setting some boundaries. After all, you can’t have your xbox and your sister too!
This is a nice demo that shows the capabilities of GPT3. And I’ve done at least 5 of these demo chatbots so far and all my friends have been thrilled. Anyway none would bend online.
If you can send me a link, I would love to test your chatbot.
Hi Dave, I am new to the community and am very impressed with your model. I am trying to build something similar and would love to see your fine-tuning data! LMK if you want my email as well. best, -david
