Best ways to prevent tokenization of unique identifier (Product SKU codes) in the training datasets when fine-tuning the model

Prompting
, , ,
1

I have successfully fine-tuned an OpenAI model but its performance is weak and it is only complete some basic answers, now i want to pass SKU codes of product as a unique identifier or as a single token to improve its performance

Can someone help me understand how can i convert unique IDs or SKU codes into single tokens, which can be used inside my training dataset files?

Here is an example prompt and completion inside my training file:
prompt: “can you provide me with 3 sku codes for dark grey products that require no assembly?”
completion: " TBAC2899, TBQC2819, ACAC2811"

I do have many basic prompts and basic completion answers which work very well,
Also , i’ve associated product SKU codes with its features and other product properties in the training set as separate prompts and completions which works well

But when i ask fine tuned DaVinci to respond back with list of SKU codes it hallucinates the answers with incorrect sku codes that is why i want to pass it as single token…

How my SKU codes are tokenized right now:

image
image719×579 22.5 KB

2

This is not possible. You can’t change the way tokenisation works.

3

Taking a step back, what’s your actual end-goal? Answering questions based on product info?

4

A bit of everything… I am not going after factual answers, i know, to do that it is better to use embeddings…

I am trying to teach fine tuned model how it can co-relate different SKU codes and then feeding basic information about the products as well so it can mimic that process for other SKU as well in the training datasets