John Schulman on how posttraining tames the shoggoth, and the nature of the progress to come…
Timestamps:
00:00:00 Pre-training, post-training, and future capabilities
00:17:21 Plan for AGI 2025
00:29:43 Teaching models to reason
00:41:14 The Road to ChatGPT
00:52:37 What makes for a good RL researcher?
01:01:22 Keeping humans in the loop
01:15:39 State of research, plateaus, and moats
I’m pretty skeptical of what these folks are dishing out.
Don’t get me wrong, gpt-4o is huge for sure, but it’s huge in that it will accelerate proliferation horizontally, but I see little indication of vertical progress and even some degradation.
90% of what we do all day is auto-pilot, and sure, getting to that with ChatGPT was a huge win. But there are so many endless stories of AI or ML getting to 90% and then just eeking out marginal gains thereafter.
I recognize that avatar! 
Yeah, I’m fairly certain they have something even better they’re not showing us yet. No proof or anything, but a gut feeling. Maybe the next version warned not to release itself yet? Heh.
I remember when GPT-2 dropped (and we got the weights) … OAI only released the smallest model… for safety! Kind of a marketing ploy if you ask me, but eventually they released the biggest GPT-2 model and world didn’t self-destruct…
We have had problems with other AI tech (like voice recently)… and scammers ARE going nuts with AI tech all over the place.
We’ll just have to wait and see, I guess, as they drip it out to us peons!
Thanks for the comment. Trying to stir up more good convos here, and your input is appreciated.
Thanks for sharing Paul! The time stamps are great too!
Folks have been saying that on day 1 when they hyped how it could build an app from a prompt and a napkin. Still waiting for that.
4GLs have been around for 50 years, and frankly, they are more capable than ChatGPT.
GPT is a killer app for brainstorming and learning about stuff, no question, but AGI is more than a couple of years off.
Interesting considerations. I think once text reaches a plateau, VTubers data might give that boost of body language, eye movement and so on - but I think that might be a novel approach we’re not even close to getting to.
Anyways, here’s some interesting data from the simple-eval repository on github:
It’s captain obvious, but in order to get funding for a startup you have to make some very outrageous promises.
VC knows these promises are outrageous - without a doubt, but they gamble looking for the unicorns. It’s the model.
The failure rate for startups is around 90%
I’m not saying OpenAI will fail, just that we need to take these things with a grain of salt.
Their job, right now, is to talk aspirationally.
If we go by the definition laid out here - https://arxiv.org/pdf/2311.02462 - than ‘competent’ AGI (the next level above emergent, which we’re already at), should be able to perform better than 50% of “skilled humans” (a term they avoid defining, amusingly enough).
That’s fine, but here’s the thing - a skilled human can be given a long horizon task and they can go away and work on it with minor course corrections. It won’t change the world probably, but any skilled human should be able to independently generate something large in scope which solves the problem the org needs solving.
Eg: I could tell a 50% skilled human “I want you to go create a marketing site for a new product. Use our current marketing site for product X as a template. Bob, who made it, can let you know where everything is to re-use so ask him if you need some help on that. If you have any other questions, let me know. I’ll check in with you in a week.”
Oh, and btw, the 50% skilled human will very likely be skilled in the latest GPT technology. I mean, for real, prompting an AI is something 50% skilled humans these days are going to learn first before any other skill.
That said, I can see a world in 2027 where there are single purpose solutions and frameworks that can use GPT to do the above - for a price. And there will be such a wide variety of them that when taken together as a whole, it could be considered ‘Competent AGI’.
But if that’s the bar, I’m afraid we passed that quite a long time ago, before GPT even arrived. These sorts of platforms have existed since the invention of 4GLs.
Still, I suppose there could be a meta agent that can thoughtfully select from these products, let you know what they’re capable of, and even drive them - and you could get things done that way. It’d be a very piecemeal way to get to ‘Competent AGI’, but it could maybe cover 80% of tasks that way.
OAI was kind of going down that direction with plugins, but I guess being commodified like that never really excites anyone seriously.
