Text 2 SQL Finetuning and metadata

API
1

I am currently working on a text2sql project. To avoid sending a large text of metadata as a part of the prompt as context, I created a set of question and answer pairs as fine tuning training data. I included all SQL metadata along with every question and the correct SQL as the answer in the training data

My assumption was while learning the pattern of SQL for given questions and the metadata, it will “remember” the metadata and so, when I use this finetuned model for inference, I don’t have to send the metadata every time

But that doesn’t seem to be the case. Unless I give the SQL metadata, the SQL generated is not useful with generic names for objects. LLM appears to learn the patterns of SQL but doesnt remember the metadata

So the question is, whether this is the expected behavior? I tried finetuning mistral and Llama 3 but neither of them gave expected results.

Before trying OpenAI, wanted to have a little more insight.

2

Hi there and welcome to the Forum!

Yes, what you are observing is expected behaviour. During the fine-tuning the model does not actually learn facts/knowledge, it merely picks up certain patterns. Hence, when you consume your fine-tuned model you indeed have to include the metadata every time just like you did in the training data.

1 Like
3

Thanks.

it is one of those cases where my whole metadata, created as a detailed semantic layer, comes up at about 50K to 60K tokens. I am looking at millions of tokens per day per user. Looks like I need to use some mini models to keep the costs down. But SQL generation capabilities of these mini models are not that great…