Hi Im trying to fine-tune the gpt3.5 model and my dataset is a transcript from a call, so there is a lot of back and fourth how would I structure this in a way so my fine-tuned model should be able to generate a script based on these transcripts.
Hi and welcome to the developer forum!
Can you explain a little more regarding what you want the fine tuned model to do?
When you say script, do you mean like a movie script? or a code script?
Hi thank you! Yeah I want it to generate a sales script based on all the success full transcripts I would pass into it. The script is a conversation no code all words.
At first thought, this is a super complex task, you’re wanting the AI to basically generate an idealised script optimising for the best interaction at any given point in the conversation.
The thing I would liken this to is a Sales “Chess” computer, where it is trying to find the best move at any stage of the game, but based on your internal conversation training data… This does not immediately jump out at me as being a large language model optimised task.
I’m not saying that this is not not totally solvable with an LLM, but it feels like a large project, 1000+ hours if you were to ask me for an initial guestimate of time for a minimum viable R&D tool.
I think you would have to combine some human feedback system in here at some point, so you’d need to score the transcripts of conversations to assess the success of it so far, perhaps starting with conversations that achieved an ideal closure and then using those, and perhaps putting them in a vector database for a similarity search… not super simple for sure.
Lets see if anyone else perhaps has a use case more similar or may have worked on systems like this.
Okay no worries. At the moment Im just doing some discovery my other option was to turn these scripts into embeddings with llama index make a suitable prompt for a LLM and ask to generate a script for a sales call. I feel like I wasn’t clear enough with what I mean, I dont necessarily want it to generate a script for a person to follow word for word more so a general guide on certain things to say that has worked for pervious people etc let the LLM be able to spot patterns and take note on the type of language the sales people have used and to take that into account when generating a script/general guide. Does that make more sense or do you want me to elaborate more on certain parts
It would help if you could describe a typical end users interaction flow with the application, i.e. name a goal and then the steps the user would do and the ideal responses an app would make to the users requests.
A goal would be to generate a script based on the style and format of scripts that fine-tuned the model. The user would input. Im trying to sell X to client Y can you generate me a script for this. The LLM would then generate a script which follows the general pattern of the transcripts which hopefully should have patterns in them which have lead to sales. Ideally the script would contain info relevant to the product and the language should be in a professional way.
I feel this may be a use case more directed towards embeddings then fine-tuning. Agree?
Yes, as the open ended nature of selling X to Y is so vastly wide, I doubt you can train on it, i.e. selling centrifuges to a laboratory head is wildly different to selling ice cream to a retail store owner.
No worries. Thank you for your help and advice! Enjoy the rest of your day.
1 Like
Hi Hoggarth. I skimmed the conversation and it’s a very interesting (and worthwhile) challenge that you present. I’d encourage you not to give up, as I think it’s solvable. What you likely need to do is to add an intermediate step: take all the transcripts that you have, and use the model to generate high level approaches for each of them (playbook strategies, or scripts/templates that serve as shorthand descriptions of the approaches being used), and perhaps also supplement it with contextual information about when and why these particular approaches tend to work. Then, you can take your sales situation and context and match it (maybe with vectors) to the most relevant “playbook” (or script), which in turn will have an association with relevant transcripts. Then you can fine tune a model (or multiple) model based on whatever parts of the system you want to make economical (for example a fine tuned model that asks the user for the context/situation of the sale, and then provides them with a playbook/script). Or, maybe you want to focus on a fine tuned model that can generate fictional scripts relevant to your playbook/scripts to help illustrate how to do it (but trained on your actual transcripts).
Sounds like an interesting project in any case. Let me know if you need any further help with it as I’m keen to explore fine tuning with the new 3.5 model. Previously I fine tuning helpful, but it only with a HUGE number of examples (40,000+ Q&A sets is where I started to see big improvements). But I’m guessing that the threshold for 3.5 will be far far lower since it’s essentially also a pre-trained model rather than a base one.