About fine tuning - Your opinion?

[curt.kennedy]

My attention comment wasn’t about anything you said recently @Foxalabs, the gears started turning when I read this comment here:

@_j

highlights maintaining Q_A style training attention within multi-question large context tune

But had more extensive comments over in this thread:

Discussion thread for "Foundational must read GPT/LLM papers"

@curt.kennedy Since I am thinking the LLM acts as a filter, and filters take big things with lots of information and bandwidth, and create smaller things with less bandwidth, I’m still heavily biased towards BIG data in and little data out approach, at least philosophically, based on past experiences and intuition. I’ll have to think about this, it’s very intriguing. It’s the complete opposite of what I’ve seen - big seems to fail as attention seems to drift very painfully, especially when…

I see a lot of attention dilution and worries, which I admit are real, but unless you are using GPT-4-32k, you probably don’t have to worry about it.

Maybe the only types of domains where you could worry about attention, is detailed coding syntax things, or similar, where you may want to reduce the tokens.

I’m just thinking if “it takes me more to concentrate on it, then reduce the tokens for the model to also concentrate on it”. I don’t have any proof this intuition is true or not, because I assume “rare things have less training than common things” in the training stages of these models.

Of course OpenAI could have trained the hell out of coding in these models, and skimmed natural language. I don’t really know. But the question is, is there more natural language than code? I don’t know, there is a lot of code online in GitHub, maybe more than all the blogs combined.