How to fine-tune a bot to accurately analyze conversation for sentiment based on pre-existing tests?

[tventura94]

So for instance, if I wanted to train a bot to give a quantitative analysis of a summary of a conversation, but based on existing tests like the Beck Depression Inventory. So in example, you’d train the bot to essentially accurately give a quantiative analysis of a conversation according to a pre existing test like the beck depression inventory.

[_j]

@tventura94

I am seeking guidance on how to effectively fine-tune an AI bot for sentiment analysis, specifically using established psychological tests as a benchmark. My objective is to create a bot that can provide a quantitative analysis of a conversation’s sentiment, mirroring the scoring system of these tests.

Here are my key inquiries:

1. Establishing the Foundation: What are the initial steps in training a bot to perform sentiment analysis? What resources or datasets would be most beneficial for this task?
2. Incorporating Psychological Tests: How can I integrate established psychological tests, such as the Beck Depression Inventory, into the bot’s training? Is there a recommended approach to adapting these tests for AI use?
3. Quantitative Analysis: How can I ensure that the bot’s analysis is quantitative, reflecting the scoring system of the chosen psychological test? What strategies can be used to verify the accuracy of the bot’s sentiment analysis?
4. Example Scenario: For instance, if a conversation summary is inputted into the bot, how can it accurately score the sentiment based on the Beck Depression Inventory?

I appreciate any insights or resources that could guide me in this endeavor.

Quick answer:

Fine-tuning is providing the types of inputs that you wish to present to the AI in practice, and then the ideal outputs that it should construct.

That is it. With hundreds or thousands of examples so that hopefully the AI learns not just the behavior and the form, but also the criteria.

In the case you put forth, just the fine-tune examples are not likely to be enough to then reduce to a simple prompt. You’ll likely need to train on and then use a system prompt nearly of the same quality that can perform on its own that can provide the AI the logic behind the examples.

Conclusion: Try thousands of prompted tokens on regular AI models before you attempt to fine-tune on something that requires logical processing or concrete knowledge hard to deduce from examples. The fine tune model costs eight times as much beyond the preparation of training materials and the training costs per token.

---

Point to ponder

• can GPT4 do this with all the prompting and programming in the world? Can it then provide you the example outputs of the quality that you could train on.
• can the lesser cognitive ability of gpt-3.5 models perform the same logic and judging procedures, even after a fine-tune?

OpenAI’s elaborate presentation on making an AI fine-tune with internal tuning resources … that can say “good” or “bad”:

Using GPT-4 for content moderation