Title: 8 Predictions for the Era of Continual Learning
Dwarkesh Podcast
Subscribe Sign in
Playback speed
1×
Subtitles
English
Share post
Share post at current time
Share from 0:00
0:00
/
8:37
S2
SPEAKER 2
0:00 So I’ve explained elsewhere why I think actual continual learning is needed.I don’t think you can have AIs that perform whole jobs as competently as humans if they are forced to just write marked-on files from session to session.Just to give an illustrative example,imagine if this is the way that students learn to play the saxophone.
So you have one student,he’s never played the saxophone before.He goes into the music hall,he tries to play it,of course it’s his first time,so he fails.And he writes down a bunch of notes about what went wrong.And there’s the next student who’s waiting outside the music hall.
He comes in,he reads all these notes.He’s also never played,so of course I don’t think there’s any sequence of text they could write to each other that would allow the subsequent student to just nail the saxophone Okay,so what changes once we have actual continual learning?
One,I think that a lot of proposals that have been put forward about regulating AI assume that you train a model and then you deploy.And therefore,if you run a bunch of checks on the model before it is deployed,we can make sure that it’s not going to aid in cyber attacks or do something crazy.
I don’t think this assumption necessarily makes sense in the future.And this is one of the many reasons I’m actually kind of worried about locking in some kind of If that happens,You could potentially be locking in an archaic and potentially counterproductive approach to dealing with the threats from AI.
To the extent the government wants some way to do some kind of safety evaluation on model providers,I think it would make more sense to do monthly or quarterly risk inspections rather than trying to single out some special moment that occurs after training is done but before deployment begins,
because that will not be a meaningfully distinct category in the future.How the labs do technical alignment would probably totally need to change.Right now,a lot of research is focused on the question of how we make sure that a frozen set of weights behaves well during deployment.
But I’m not aware of much research on the question of how we make it so that Even with constant weight updates,the AI system never falls prey to jailbreaks or changes into a deceptive or evil persona.And if AIs are consolidating learnings between users as well,how do we prevent users from injecting backdoors or some kind
of malicious inclination In some sense,this is actually kind of what the human alignment problem is,right?Humans improve in a self-directed way.If you have kids,I don’t have kids,but I imagine this is what happens.If you have kids,they go out,they learn new things.Sometimes they go crazy.
Three,the diversity of AI minds will increase.Right now,there are less than five prominent AI minds,by which I mean the base models,which are served to millions or hundreds of millions or billions of users at once.And they’re all quite similar to each other,
by the way,because they’ve also been all trained on roughly the same data.We’re learning from experience.And that experience is different between not only different AI companies,but also between different instances of the same AI model.We can actually see a lot of diversity come out the other end in this world.
And this would be,I think,a net good outcome.I think one of the things to worry about in the future is just having this monolithic singleton that’s quite boring.A world where we have continual learning would hopefully be more interesting than the mode collapse of different models we see in the world right now.
When deployment becomes part of training,the returns to being ahead in the AI race accelerate.Because if you have the best model and more people are using your AI for more complicated and useful work,and as a result they’re giving lots of feedback that can integrate beyond the session window,then your model will become even smarter.Fight.
If the model learns mainly from deployment,then labs will feel a lot of pressure to deploy their smartest models earlier.Anthropic has reportedly been using Mythos internally since February,but it only shipped the model to the public in June.In the regime with actual continual learning,this kind of thing would just 6.
Continual learning will create a clear mode for the leading AI labs that they currently lack.Many people have been asking,how will the AI labs actually make money?I have been asking this.When I had Dario on the podcast,I asked him this question.And he made the analogy to cloud providers.And he made the point,look,
the cloud providers are offering many undifferentiated services,but they’re earning They’re doing high profit margins nonetheless.You will have noticed this if you look at Amazon or Google’s quarterly earnings,they’re doing just fine.But the reason that the cloud margins are so high is that it’s really time consuming and expensive to switch from one cloud to another.
Currently,there’s nothing that stops But once we have actual continual learning,and the model you’re working with is actually getting better as it interacts with you from session to session,then there are actually pretty significant switching costs.If you want to change the AI that you’re using,
you basically have to fire an employee that has accumulated months of context on your organization,and you replace them with a very fresh,very unexperienced new intern that you got to retrain from scratch.And once you have this kind of lock-in,model providers can demand pretty hefty margins.Seven.
Of course,enterprises will be wise to this kind of dynamic.They will try to avoid this kind of lock-in.But what if the choice is that you either get locked into a model provider or you lose out on this super valuable feature where your model improves for you from session to session?
If real usage ends If this ends up being the main way the models improve,then the AI labs may subsidize users and enterprises which allow the model to train on their sessions.This is already happening if you look at the kinds of deals that are offered to new users of coding products.
This is very similar to why Google gives away search.And conversely,the labs may say that any enterprise that refuses to let them train on the sessions can’t have access to the very best models.With both carrots and sticks,the labs can do a lot to get users to allow AIs to learn from experience.Now,of course,
I’m glossing over the fact that there’s a difference AI training already has large economies of scale.You get to amortize All this expensive trading across more users.And you see the evidence for this in the fact that the lab revenues are increasing far faster than their compute.
But continual learning may also lead to economies of scale in inference for end users,namely from batching.You might have seen my episode But if per company instructions require full weight updates rather than living in low rank adapters,there’s huge advantages from batching.Back of the envelope math suggests that the optimal inference batch size for
a sparse model like DeepSeq v3 is more than 2400 concurrent sequences being generated at once.If you don’t do this,then you’re under utilizing your compute.And if you want to understand why,again,I highly recommend that episode with Reiner on inference economics.But anyways,the point here is,
is that a given set of weights is only served efficiently when thousands of sequences are being decoded against it all at once.A large company with lots of employees and agents who are doing lots of different kinds of things can very efficiently serve their weight fork.
Whereas an individual user who’s only running a batch size one may suffer more than two orders of magnitude worse efficiency on their compute.So the economics of serving personalized weights strongly favor big organizations.Obviously,plenty more will have changed by the time that continual learning actually works.
And the most important changes are probably the ones that are hardest to anticipate in advance.But the ones above seem kind of clear even now.This was a narration of a blog that I also published on my website.Go check it out at dwarkesh.com.Otherwise,I will see you on the next podcast.
115
19
8
8 Predictions for the Era of Continual Learning
Locking in AI safety regulation now is a mistake.
Aug 07, 2026
115
19
8
Share
Transcript
I have explained elsewhere why I think continual learning is needed. I don’t think you can have AIs that perform whole jobs as competently as humans if they’re forced to just write Markdown files from session to session.
To give an illustrative example, imagine if the way students had to learn to play the saxophone is that one student tries to play it from a cold start, then after her first session, writes down a bunch of notes, then the next student waiting outside the music hall who’s also never played the saxophone reads all their notes before trying to play, and so on. Even if you had an infinite sequence of saxophone-virgin students waiting outside the studio that could write notes to the next guy, there’s no sequence of text they could write together that would allow the Nth student outside to play proficiently on their first try. At some point, you have to accumulate the experience into the brain. I think the same will be true about a lot of skills and knowledge that we’ll want AIs to learn in all the different workplaces they find themselves deployed in.
Okay, so what changes about AI once we have continual learning?
-
A lot of the proposals that have been put forward for regulating AI assume that you train a model, and then you deploy it. And therefore if we run a bunch of checks before the model is deployed, then we can make sure that it’s not going to aid in cyber attacks or recursive self improvement. But what if the base model is getting updated every single day based on the millions of sessions of work it does? This is one of many reasons why I think it’s unwise to lock in some kind of regulatory safety regime right now. We simply don’t know what kind of technology we’re going to be looking at even in a year, let alone in five years or ten years, and we’d be entrenching an archaic and potentially counterproductive approach to dealing with the threats from AI. To the extent that the government really wants to do some kind of safety evaluations on model providers, it would make more sense to do monthly or quarterly risk inspections rather than singling out some special moment that occurs after training is done and before deployment begins, because that will not be a meaningfully distinct category in the future.
-
How the labs do technical alignment would need to totally change. Almost all current techniques are focused on the problem of how we make it so that a frozen set of weights behaves well during deployment. I’m not aware of much research on the question of how to guarantee that, even with constant weight updates, the AI system never falls prey to jailbreaks or changes into a deceptive or evil persona. And if AIs are agglomerating learnings between users as well, how do you prevent users from injecting backdoors or some kind of malicious inclination into the base model? In some sense it is actually closer to the human alignment problem - your kids go out and learn new things, sometimes get one-shotted by crazy ideologies or drugs or something - but you hope you’ve given them enough common sense and basic values to improve as people in a self-directed way, without ending up with some super weird and misanthropic beliefs.
-
The diversity of AI minds will increase. Right now, there are <5 prominent AI minds, and they are all quite similar to each other on account of being trained on roughly the same data. But if AIs are learning from experience, and that experience is different between different AIs, we could see actually different AIs come out the other end.
-
When deployment becomes part of training, the returns to being ahead will accelerate. If you have the best model, and more people use your AI for more complicated and useful work, and give it lots of feedback that it can integrate beyond the session window, then your model will become even smarter.
-
If the model learns mainly from deployment, labs will feel the pressure to deploy their smartest model earlier. Anthropic has been using Mythos internally since February. But it only shipped this model to the public in June. In a continual learning regime, a four-month internal/external gap means ceding four months of deployment learning. Your competitor who ships earlier might be worse than yours on release day, but it gets to use a lot more real world experience to get better.
-
Continual learning will create a clear moat for leading AI labs that they currently lack. Many people have been asking, how will the AI labs actually make money? When I asked Dario this question on my podcast, he made the analogy to cloud providers, who also offer many undifferentiated services, but earn high profit margins on them nonetheless. But the reason cloud margins are high is that it’s really time-consuming and expensive to switch from one cloud to another. But currently there is no switching cost for AI models. There’s nothing that’s preventing me from starting a software repository with Codex and then finishing it with Claude Code. But once we get continual learning and the model you’re working with is actually getting better as it interacts with you from session to session, then there are actually pretty significant switching costs. If you want to change what AI you are using, you basically have to fire an employee that has months of context on your organization and replace them with a fresh one that you have to retrain from scratch. Once you’re locked in like this, model providers can demand pretty hefty margins.
-
Enterprises will be wise to this. They will try to avoid this kind of lock-in. But what if the choice is that you either get locked into a model provider, or you lose out on this super valuable feature where the model improves for you session after session? If real usage ends up becoming the main way models improve, then labs may subsidize users and enterprises which allow the model to train on their sessions, especially on hard economically important work. Just the same way that Google gives away search. And conversely, the labs may say that any enterprise that refuses to let them train on its sessions can’t have access to the very best models. With both carrots and sticks, the labs will try to get their users to allow AIs to learn from experience. I’m glossing over the fact that there’s a difference between updating one user’s set of weights, and pooling all these different weight forks back into the main model, and the latter may be technically more challenging, but in due time that too will be solved.
-
AI training already has large economies of scale (the theoretical reason to expect this is that you can amortize your expensive training across more users, and the practical evidence is that so far, lab revenues have increased faster than their compute). But continual learning may also lead to economies of scale in inference for end users, namely from batching, if per-company information requires full weight updates rather than living in low rank adapters. Back-of-the-envelope math suggests that the optimal inference batch size for a sparse model like DeepSeek V3 is > 2,400 (that is to say that unless your model is concurrently generating that many sequences at once, you’re underutilizing your compute). If you want to understand why, go watch my full episode with Reiner Pope on inference economics. But anyway, the point here is that a given set of weights is only served efficiently when thousands of sequences are decoding against it at once. A large company whose employees and agents generate that much concurrent traffic can efficiently serve their continually-updated weight fork; an individual user serving themself at batch size 1 might suffer a 100x+ compute efficiency penalty. So the economics of serving personalized weights strongly favor big organizations.
Plenty more will have changed by the time continual learning works, and the most important changes are probably the ones hardest to anticipate. But the ones above seem clear even now.
Discussion about this video
Comments Restacks
I follow Silicon Valley and AI culture fairly closely, but from a distance. I’m not immersed in it. So when I saw the title “The Era of Continual Learning,” my immediate assumption was that the piece would be about continual learning in humans. I was genuinely surprised to discover that “learning” here meant AI learning, apparently without the need to specify it.
That surprise ended up shaping how I read the rest of the essay.
The distinction between learning through context/Markdown files and learning through weight updates seems, in human terms, a lot like the distinction between explicit and implicit knowledge. The saxophone analogy makes a compelling case for the latter. You can write increasingly perfect instructions for playing saxophone, but at some point the learning has to make its way into the body. You can’t reason your way through every movement while playing a beautiful solo.
But I’m less sure that the same hierarchy applies to thinking and decision making. Explicit knowledge has an important property: it is inspectable. If part of my reasoning lives in a Markdown file, I can interrogate it, challenge its assumptions, trace where it came from, and rewrite it. Once learning has been absorbed into the “weights,” its influence may be deeper and more fluid, but also much harder to see.
Which brought me back to my initial surprise at the title. Dwarkesh himself seems to have been “continually trained” by a particular culture and intellectual environment to the point that “continual learning” can naturally mean AI continual learning without distinguishing it from human continual learning. That assumption isn’t sitting in a Markdown file somewhere. It has become part of the weights.
That seems like both the power and the danger of the kind of learning the essay describes.
Indeed. Importing human features of intelligence will bring along multiple unintended consequences. Im not saying continual learning AI models shouldn’t be a goal, but it will change things in unexpected and perhaps undesirable ways.
The idea of a continuous learning AI is powerful, but it needs judgement, not just feedback. How would the AI know if the feedback it receives is from an expert, a novice, a bad actor, or just another AI producing slop? Dwarkesh I think the Frontier labs are doomed, in a way, unless they can verticalize their spikey models in many domains. Ultimately “AI” has to fragment into different domains, otherwise it seems that the Frontier models start consuming the enormous amount of slop created by other AI’s and they just get dumber, fuzzier, and less useful.
8hEdited
I just listened to the podcast again and there’s a big misconception here. He seems to believe that the frontier labs are generating knowledge. They’re not. They’re just accumulating tokens and trying to arrange them in a way that people respond positively.
The reason they’re making so much money is that they’re doing an amazing job of accumulating and simplifying things on the Internet that are hard to find. So they are actually very very sophisticated search engines. And that when you had a little bit of logic, it’s very powerful for booking a flight or fixing a piece of software or looking up information from the government where it’s very hard to find.
I think if we move to the vertical AI where these systems are actually intelligent and specific domains they go in very very different directions. I know in our particular case the AI that we have built is very unique and different from the frontier models because it has unique information and knowledge and experience and examples that do not exist on the public Internet. And that has to be where this market goes.
So this idea that the AI itself is going to learn feels to me to be the wrong direction. It’s the humans that are going to teach the AI not the AI that’s going to learn on its own.
Remember that no matter how hard these models try to learn at least 50 to 75% of their content is going to be exaggerations or opinions or non-truths. Except if they’re focused only on an narrow domain. And that would lead you to believe that the frontier labs have to purchase content or else become big search engine, like Google.
Dwarkesh, this analysis is perhaps too focused on current monolithic “frontier-class” models as far as I can see. How do highly-capable open-weight models tuned on localized data and focused specifically on local conditions fit into this scenario?
3hEdited
Not having continual learning is a safety feature. Advocating for it is advocating for removal of safety.
I’m not aware of much research on the question of how to guarantee that, even with constant weight updates, the AI system never falls prey to jailbreaks or changes into a deceptive or evil persona.
If Putin is a good leader in 2002, how do you guarantee he won’t want to start a big war that kills a million or two people 20 years later? I can’t rule out that such a guarantee is possible, but I wouldn’t expect it, and we already can’t fully guarantee safety in the frozen-weight case, e.g. it’s not provable that there’s no general jailbreak for a given model (I’d bet that there is one, it’s just hard to find).
Can’t say I’m not curious what could be done with a small blob of dynamic weights that exist alongside the big frozen weights, though… like those little blobs that fall off GLaDOS…
This is all nonsense. One can not control langauge with language. Give the LLM a terminal attractor and define the state space and get out of the way, Just make sure the attractor isn’t more. More is undefined.
I don’t think your argument here supports your headline about AI regulation. Time to deployment of CL is unknown, possibly never. Agree with your first two points: CL could change both the details of regulation and of alignment during development. It definitely makes the AI more human-like in the sense that its behavior reflects ongoing experience, with the possibility of going towards the Dark Side.
I think your later points about lockin apply given the assumption that what the AI learns about is you! It’s not clear this absolutely must be done by weight updates. I could imagine developing better and better summaries of past conversations available to a non-CL AI that could help it have access to relevant information about you without learning in the full sense. I’m sure the labs are all thinking about this.
Finally, your point about batching caused me to wonder if a weight updating AI would have sequential dependencies that would preclude batching! That could cause some hiccups.
13hEdited
Love all your content Dwarkesh thanks!
I’m not feeling the saxophone metaphor. Playing saxophone requires physical procedural memory. Agreed that’s something you can’t pass effectively through notes.
But now imagine instead a sequence of knowledge workers in a marketing job or an accounting job at a desk in an office. Each one learns on the job for say an hour and then writes extremely detailed notes up to a million tokens long (several books full) for the next one. And now imagine the next worker can read and fully absorb books full of notes almost perfectly in seconds and apply them in their work.
Suddenly continuous learning doesn’t sound so interactible, does it?
14hEdited
Another way to think about the saxophone learning problem might be that AIs (or humans for that matter) don’t have the right symbolic language (or model) to describe that kind of physical learning, hence it cannot be written down effectively. It would be like trying to explain/solve a hard math/physics problem using pure english without any symbolic math.
One way humans have gotten smarter at understanding nature over time is by developing better language capabilities, that let us think and reason about concepts by writing them down symbolically. So instead of continual learning the way you are describing it, the other way to solve this problem might be for AIs to develop a language for symbolically describing this kind of physical learning and then sharing the learning. Us humans have never had to do that because we can interact with the physical world and “learn” it slowly by doing. AI cannot (at least currently).
A corollary of the same observation is that this is also the reason why AI is so good at coding and math currently. Because we do have a great language for writing down, reasoning and passing the understanding symbolically.
I think inventing the right symbolic language might be the missing link here.
I hadn’t thought much about the switching cost. Once a model learns how a company works, leaving the provider could mean starting over. Companies will want a way to take that history with them.
Continual learning is what turns the model from a depreciating asset into a compounding one, and that’s where the economics get interesting. The initial training is the cheap, one-time part; what you feed it afterward becomes the scarce resource, because that flow is exactly what only you can verify and curate. Generation stays free, but the ongoing data is the tax.
Prediction 5 also changes the viability of voluntary safety holds. In your example of Anthropic holding Mythos for about four months, that delay becomes costly once deployed models learn from user sessions and compound their lead. If competitors gain that lead by shipping earlier, “we held it for safety” becomes a first-mover disadvantage, and continual-learning economics will select against labs that impose such holds.
1dEdited
OK, I’ll leave “continual learning” as a term to be implicitly defined (as are ALL words) in what you say (won’t follow the link you provided, what can be said can be said clearly - and necessarily briefly) - as I read the article - and will be adding to this comment continually - but, tbh, have low expectations - it’s a process of human intelligence and character - and who understands that? You guessed it.
1: You’re still stuck with static (and highly inefficient) models with largely static “knowledge” and now imagine them learning continually. Very speculative - though you might of course model/implement how humans learn (see above) - and don’t restrict your fact-finding to yourself. Oh, and how about regular risk-inspections of you and other individuals?
2: Ha, ha, you’re on the right track - you have no control over how your kids will turn out - and an AI is incomparably faster - so also faster in “breaking bad”😂
3: You didn’t get the distinction between a species and an individual, did you?
4: Same challenge as in modern (geo)politics linked to social media: It’s all well and good to have a growing following - but when do you leverage (e.g. monetize) them? To outcompete all others, of course - which is the basic premise of the Techworld - and of predators.
5: Yeah, right, humans can’t be made battle-ready in less than, say (going by child-soldiers) 10 - need to improve on that.
6: Ha, ha, good one - so how does intelligence (think human) become succesful, so earn money and get at the levers of power?
- SURE, lock-in is well-demonstrated by autocracy - worked for millennia one it started - and now you can drastically supercharge it👍
8: Sure sure. sure - benefits of the successful always trickle down to the expendables - not really proven - but a basic premise.
Was that it?😮 Maybe ask an AI whether this makes sense - but of course you must give it some basic axioms as guardrails/optiization-criteria.
I’d read the regulation point as an argument for a different cadence, not for waiting. Monthly risk inspections are still a regime, and other industries already supervise systems that keep changing under them: drugs get post-market surveillance, aircraft get continued airworthiness. What dissolves is the checkpoint, not the reason to look.
I’d expect that continual learning might make certain types of progress easier within a given operating environment, but I’m a bit skeptical that it’s necessary and some of the problems you identified seem like ~non-starters. Having a stateless model which can be configured via the context window seems strictly superior (assuming equivalent performance).
Staying in the saxophone example, imagine if it were possible to hill climb on students+notes to the point where you could give the Nth student some notes and instill in them the ability to play the saxophone. That’s much better! Take an arbitrary student and imbue them with saxophone skills. More amenable to composition, and easier to identify and resolve cases where skill compositions result in misalignment. (In addition to avoiding the problems you mentioned above.)
And tbh, the more realistic analogy here would be passing some multimodal sensory sequence to the next student. Experiences for LLMs happen in context windows, ie if an agent were to be able to “learn” some skill within a given context window, then necessarily the activations are reachable through some sequence of tokens. And there’s probably some shorter sequence that would provide the same capabilities.
The saxophone analogy is a poor one because learning the saxophone is something intuitive that you can’t learn simply from reading. In most white collar jobs, it’s probably quite possible to operate effectively if you’re smart enough and you’ve read examples of previous cases the company has encountered. A 1-million token context window is enough for that.
I’d say there’s some value in the analogy, but it does seem cramped. It would be one thing if models could write down just the equivalent of musical notes. But they can do much more than that. They can write anything that can be communicated symbolically. So, to continue the analogy, they can write down “hold it so high,” “angle it this way,” “place your fingers like so,” “blow into the mouthpiece with this intensity.” It is not intuitively obvious to me that a genuinely useful set of instructions could not be rendered compactly enough to permit the models to make progress.
Still, dynamically updated weights would be much better. Does anyone think that is close? (Honest question. I have no idea.)
1dEdited
Nice read! I like the saxophone analogy for how AI tries to learn complex / subjective tasks right now.
Are the frontier labs or any startups actually working on achieving continual learning by updating model weights? All of the startups I am aware of (including my own) are trying to solve adaptive AI problem by improving the model’s ability to store and retrieve relevant information during the tasks. The strategy of dynamically creating RL environments to use for improvement is also promising and being pursued. If you are correct that modifying model weights will ultimately trump all these methods then that means a lot of startups are trying to do the stop-gap solution. I am not 100% convinced that modifying weights is necessary — I think for all text-based tasks meaningful results can be achieved without changing the base model.
Dwarkesh Podcast
Deeply researched interviews
Deeply researched interviews
Subscribe
Listen on
Substack App
Apple Podcasts
Spotify
YouTube
RSS Feed
Appears in episode
Recent Episodes
Why smarter AI models could drive up compute prices 10x
Aug 3•Dwarkesh Patel
Adam Brown – A deep but accessible introduction to general relativity
Jul 10•Dwarkesh Patel
Grant Sanderson – AI and the future of math
Jun 30•Dwarkesh Patel
The next big breakthrough will be AIs learning on the job
Jun 26•Dwarkesh Patel
The data black hole at the center of AI
Jun 19•Dwarkesh Patel
Ada Palmer – Machiavelli is the most misunderstood thinker of all time
Jun 16•Dwarkesh Patel
Alex Imas and Phil Trammell – What remains scarce after AGI?
Jun 4•Dwarkesh Patel
Ready for more?
Subscribe
© 2026 Dwarkesh Patel · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture