Make part of the training where algorythms that are capable of lerning/neural network/GPT/etc be able to chose what training data to get more often for different training methods.
Premise 1: Some training Data is more important/useful than others. (Example: Lets hypothetically say „rbeufufbfjdj37;6:6fkfkdjdjdh“ is the least useful training Data and „[magic equasion that speeds up/improves resoning in 80% of the model]“ is the most then it would make sense for o train on the most useful thing more.
How to do that? Well just make a reward function that trains on selecting the best training data and feed it into the training more than once or let more „thorrow“ / extra training methods run over it and the worse date the opposite to save ressorces (opportunity costs).
If for example [Training Data: A] is 100% then the model could split it in two 50% 50% pieces and then if the frist one shows better scores in test metrics and performance improvements and desired improvements lets say it is 3.45% more effective then the other halv then you already have that and you can split the 50% in two 25% 25% peaces or let the modeö by its own metrics (for example by choosing topics or people or anything it selects or „wants to train on“ just like a human might lern that a weekness of his is X and then chooses a Book on X rather than Y because the hypothetical human already knows Y but would benefit from training data Y WAAAAAAY MORE.
Its relly simple, so did you already do that? Make many such systems and let them compete and make sure you dont miss systems that might be good at some domain not tested for or need time to „ramp up“ or get better faster than other systems when something improves or or gets worse, bigger, smaller, cheaper, more expensive, slower, faster etc.
Just dont lose yourself in ruling out stuff. And realize that humans in history where pretty much always wrong or not that correct and time did tell meaning nothing should be off the table to question and nothing off the table to try altho being smart, ressourceful etc is always good.
Short: Make algorythms that can select training data for example gpt should realize if the trend of something does more good after it got better and therefor hypothesise that the training data might have preciously not made sense to it but then became more useful once other premises and prerequisites of its „resonibg“ for example took shape developed.
Give it the ability to write things down kinda like „hypothesis“ and then act on those to select training data and the methods best to train on the training data.
There are many thibgs that could help even the ability to make „quick tests or have some simulation of if premise X where to be acceptet what would that do does it fit with other premises one assumes to be true“? Therefore being able to categorize things and ja…. Complex stuff
But mist importantly memory/context window unlimited would be the biggest improvement im guessing imagine a 1 200 tousand word chat long or 2 million word long account chat history and the finetuning/tailored help GPT could give.
Also make narrow wuick finetuning possible and the system more dynamic with many parts that train lern sort execute compress decompress predict and do many things at once for effitiency reasons even if it is not as capable or would have light performance decreases in the shoetterm. Also systems that themselfs get smarter over time even if things are done incorrectly or bottlenecks are built in. Also make it so setain parts in a neural network can be coped and tryed in other parts and then distilled (the connection not the main useful „crystal“ magic formula within it are slowly wittered away until the uselfuö oart raimains that way even some useful thing lerned in a totally unrelated task might be able to inföunece another part (just like human neurals supposidly can travel and constantly build and change stuff copy and execute on dna 
(kinda like building plans)…. anyway just do the training data selection for now. If it does not improve over time or have the potential to you are doing something wrong in training in my opinion (exrapolation) just do 20% or 1% of the training in my method and then compare it to a similar % not just performance but also trend% (sice if something grows faster it might catch up