NEFTune: adding noise to vector embeddings

I recently came upon a fine tuning technique that involves adding noise to embedding vectors during training. Results have been interesting on llama with reports of efficiency jumps from 29.79% to 64.69% with the inclusion of noisy embeddings.

I know hugging face allows you to pass neftune_alpha as a parameter when fine tuning and I was wondering if there was a way to do that with openAI?

2 Likes

OpenAI does not use neftune as a parameter, but you can use learning_rate_multiplier to achieve the same goal. More information over here:

https://platform.openai.com/docs/api-reference/fine-tuning/create#fine-tuning-create-hyperparameters

3 Likes

Link to paper instead of micropayment blog site:

We hypothesize that by adding noise to the embeddings at train time, the model overfits less to the specifics of the instruction-tuning dataset, such as formatting details, exact wording, and text length. Instead of collapsing to the exact instruction distribution, the model is more capable of providing answers that incorporate knowledge and behaviors of the pretrained base model.

4 Likes