QA fine-tuned chatbot not answering from the trained data but nonfactual

API
28

Ok I see your problem. Sorry being direct, but you’re still not getting the idea of fine-running nor the embedding.

1 Embedding - helps you find most relevant text or of many texts (facts only in your application)
2 Fine tuning - helps you adjust the manner and patterns used by a model to complete a given response.

Here is how I would do the thing.

1 for each company, create a file with all facts your bot needs to know about. Simple txt file would do.
2 cut your files into rather small pieces (about 2-3 paragraphs) with text similar to:
Company: Company name
Data: your facts and other related text
3 embed reach piece of text with text-search-ada-doc (to start) and save into your database along with the piece text without Company: Company name part.

Now start seeding the bot:

  1. Chose your first question (usually the most common but rather complicated):

What do you think of the company X in today’s situation?

(Chose another one if you have no means to tell your bot what the today’s situation is)

  1. Embedd that question, using text-search-ada-query (I guess it is better to have same embedding base model, but please correct me if I’m wrong)

  2. Compare the vector of the question with your facts in database to get several most relevant facts sorted by relevancy desc.

  3. Copy the text of those several facts without the Company: Company name part. And start creating your prompt:

Bot description… Bla bla bla.
Factual context: fact 1 the most relevant… Fact x the least relevant out of acceptable.
User current state: their mood
Conversion summary: bla bla bla…bla
User: What do you think of the company X in today’s situation?
Bot:

  1. Take the prompt from step 4 and paste it into the playground (adjust settings to your will)

  2. Run/replay a completion several times and save bots best answers (feel free to adjust/edit them manually before saving) to a separate file like this:
    —prompt—
    Bot description… Bla bla bla.
    Factual context: fact 1 the most relevant… Fact x the least relevant out of acceptable.
    User current state: their mood
    Conversation summary: bla bla bla…bla
    User: What do you think of the company X in today’s situation?
    Bot:
    —completions—
    Bot’s answer 1
    –
    Bot’s answer 2
    –
    Etc.

  3. Change your question and restart from step 1 until you have at least 50 questions with 2-4 answers each.

8 create a script to reformat your saved answers into seed.jsonl file with following format for each line:

{“prompt”:“Bot description… Bla bla bla.\nFactual context: fact 1 the most relevant… Fact x the least relevant out of acceptable.\nUser current state: their mood\nConversation summary: bla bla bla…bla\nUser: What do you think of the company X in today’s situation?\nBot:”,“completion”:" the saved reply.<|endoftext|>"}

Make sure it is a valid json and reach object is on its own line (or your model will “break” most likely)

  1. Once all the lines are saved it would be nice to shuffle the lines randomly to break the “flow” (use a script for that).

10.Create a fine tune using davinci as basse model and n_epoch 2

Now, populating real training data.

  1. Create an app with question input field, question submit button, reply edit field and save button.

Submitting question sends the workflow to step 2 of the previous section.

On step 4, print the created prompt on the screen and send the prompt to the API to get bot answer, put the answer into edit answer field.

  1. Edit the answer if necessary

  2. When happy, click save button. On save button your app should save exactly one line as in step 8 of previous section (make sure you save the edited answer)

  3. have fun until you get at least 1500 answers

  4. create a new fine tune “my-bot-v1-0” based on your model (again, n_epoch 2 or test)

  5. Every 1000 answers upgrade your fine tuned model from “my-bot-v1-y” to “my-bot-v1-z” by feeding new data only