When discussing Rich Sutton, the luminary of reinforcement learning, virtually everyone within the AI community is well-acquainted with his contributions.

He penned 'The Bitter Lesson,' a seminal framework that remains a cornerstone for many in comprehending AI development: In the grand scheme, general methods that scale with computational prowess generally surpass solutions heavily reliant on extensive human-crafted knowledge.

However, in a recent interview with Sequoia Capital, Sutton cast a critical eye on the prevailing large model paradigm. In his perspective, while LLMs signify a monumental scientific leap, they could also stand as a rebuttal to 'The Bitter Lesson.'

The rationale is straightforward. Today's large models, though liberated from numerous artificial constraints, are once again shackled by 'what humans already know.' Internet data is bounded, and even synthetic data is underpinned by human decisions on what merits creation.

More crucially, models still lack the continuous learning capabilities inherent in humans.

A seasoned driver with a decade of experience develops a profound intuition. Yet, the bulk of learning in today's large models transpires during pre-training and post-training phases. Once deployed, despite daily interactions with millions, their core weights remain largely static.

In Sutton's estimation, the current large model industry may have already plateaued in 'local optima': Piling on more GPUs and scaling up still yields performance gains, making it arduous for leading labs to pivot to a new path that may initially underperform.

This serves as the impetus for Sutton and his protégé Khurram Javed in establishing Oak Lab.

Oak endeavors to engineer an AI Agent capable of perpetual learning from its own experiences: Post-deployment, the model can continuously refine its understanding of the world, extract novel concepts and rules from experiences, and adapt to shifting environmental conditions.

Should the continuous learning path prove successful, what stands to be rewritten is not merely a generation of models but the entire foundational model paradigm of today's 'train-freeze-deploy' cycle.

/ 01 / The Inability to Continuously Learn: The Achilles' Heel of Large Models

Sutton's stance on large models is quite nuanced.

He does not dismiss the achievements of LLMs.

On the contrary, he regards large language models as a 'remarkable scientific breakthrough.'

Language has long been considered a stronghold of symbolic AI. Yet, neural networks, through extensive training, have acquired exceedingly complex language proficiencies.

This dovetails perfectly with Sutton's renowned 'The Bitter Lesson.' The crux of this essay can be succinctly summarized as:

Minimize the injection of human rules into machines and instead pursue learning methods that scale with computational might.

The annals of AI over the past few decades have repeatedly demonstrated that while human-designed rules often excel in the short term, they generally yield to more general methods capable of leveraging greater computational power in the long run.

AlphaGo, deep learning, and large language models all epitomize this trend.

LLMs 'consume' the entirety of the internet and continually augment computational power, leading to swift enhancements in capabilities. Hence, Sutton deems large language models one of the most triumphant manifestations of 'The Bitter Lesson.'

But here's the twist: Large models may also serve as a counterpoint to 'The Bitter Lesson.'

Because while today's large models have shed many artificial constraints, they are once again ensnared by another form of 'human knowledge':

The internet.

No matter how vast, the internet merely encompasses knowledge already documented by humans—papers, code, books, web pages, videos, chat logs...

Models can endlessly train on this data, but the real world is infinitely more complex than the internet. Sutton's assessment is unequivocal: The information in the real world vastly exceeds everything amassed on the internet to date.

This implies that solely depending on increasingly meticulous data preparation prior to training will ultimately hit a snag.

An even more critical issue ensues. Today's large model learning and operation are artificially segmented into two phases.

During training, they absorb knowledge intensively. Once deployed, their weights remain largely immutable.

They can temporarily assimilate new information through Context or retain preferences through Memory, but Sutton contends this falls far short of genuine learning.

Because genuine learning entails the system itself undergoing transformation after an experience.

For instance, after performing 1,000 surgeries, a doctor's decision-making process evolves. After driving for 10 years, a driver instinctively senses when another car might cut in.

Experience truly permeates humans. Today's large models, in contrast, resemble a frozen brain upon graduation. No matter how many years they work subsequently, they primarily rely on revisiting their graduation notes.

This is Sutton's core critique of current LLMs:

Why can the ability to form new concepts and modify internal structures only transpire during training?

If altering weights during training can engender intelligence, why can't models persist in doing so post-deployment?

/ 02 / Synthetic Data May Not Be the Panacea for Scaling

What if high-quality internet data is finite? One prevalent solution is synthetic data.

Theoretically, with sufficient computational power, data can seemingly be generated ad infinitum. But Sutton offers a candid evaluation:

This is a 'significant misstep.'

It could even become the next 'bitter lesson.' The reason is that while synthetic data appears to transcend the limitations of real data, it still conceals a human bottleneck.

Who determines what data is worth generating? Who assesses which questions are pertinent? Who defines rewards? Ultimately, it still circles back to humans.

As long as synthetic data necessitates human input to decide what is worth generating, the scalability of this approach remains constrained by human knowledge.

Sutton's underlying theory is termed the Big World Hypothesis.

This concept is actually quite straightforward. The real world is so vast that no agent can possibly master everything in advance. When a drone actually takes flight, it encounters wind, friction, motor errors, obstacles, and innumerable details never encountered in simulators.

These cannot all be incorporated into the training set prior to model deployment. Thus, Sutton argues that the world an agent confronts will always be more intricate than the agent itself. Since it's impossible to learn everything beforehand, the only viable method is to learn as you go.

This essentially inverts a core assumption of today's AI industry. Over the past few years, the objective has been to train models to be increasingly 'omniscient' before deployment.

Sutton envisions AI that is less fixated on 'knowing everything at birth.' What it truly requires is the capacity to continue evolving thereafter.

/ 03 / The Real Challenge: Acquiring New Knowledge Without Forgetting the Old

If continuous learning seems so intuitive, why don't companies like OpenAI and Anthropic simply permit models to update their weights daily?

Because if they did, models would swiftly become dysfunctional. This is the conundrum that Continual Learning has grappled with for years without fully resolving: catastrophic forgetting.

Suppose you inform a model today: 'Our company's internal project X will now uniformly be referred to as Apollo.'

The model immediately adjusts its weights based on this data. That seems advantageous. But knowledge in neural networks doesn't exist as isolated entities. A change in one weight may simultaneously impact numerous previously acquired abilities.

The likely outcome is that Apollo is remembered, but other facets are compromised.

This is why many models today prefer to cram new information into Context, RAG, or Memory rather than readily modify underlying weights.

Products like Cursor have already implemented some form of continuous updates, but the approach remains more akin to:

First amass data from thousands or even millions of users, aggregate it into a massive batch, and then train uniformly once. This suffices for public models. But for authentic personal agents, this method becomes cumbersome.

If I merely wish to impart a new habit to my AI assistant, why await data from 100,000 individuals to train collectively? More troublingly, what I desire it to learn may hold no relevance for another 100,000 people.

Thus, what Oak truly aspires to tackle is a fundamental algorithmic challenge:

Enabling a model to update itself using continuously arriving singular experiences without erasing past knowledge.

One direction Sutton suggests is termed Step-size Optimization.

Simply put, the 'plasticity' of different knowledge should not be uniform. Some facets in the model have been verified countless times and should remain highly stable. Other aspects, newly learned, should be more malleable.

Thus, every weight can possess a distinct learning speed. This closely mirrors the human brain. An adult won't overhaul decades of understanding about 'cats' after encountering a single green cat today.

But if a previously consistent pattern is discovered to have altered over several years, the brain can gradually adapt.

Even more intriguing is another method Sutton proposes: Continual Backprop.

Traditional neural networks, upon initialization, randomly generate numerous neuron weights and then commence training. Initially, there is much randomness and diversity. As training progresses, these random spaces gradually diminish, and the model's internal structure becomes increasingly rigid.

What Sutton seeks to explore is how to maintain a neural network trained for 10 or 20 years as malleable as a youthful brain. This is what Oak aims to uncover.

They believe that when training foundational models in the future, this capability should be cultivated directly.

From day one, the model should concurrently learn two things: one is knowledge. The other is how it should assimilate knowledge in the future—that is, meta-learning.

Truly continuous learning models may necessitate being retrained from scratch.

/ 04 / Oak Is Betting on the AI of Tomorrow, Beyond Large Models

If Oak were merely addressing 'model memory,' the company wouldn't garner as much attention. What renders it truly ambitious is Sutton's aspiration to link continuous learning with another capability: autonomous abstraction.

Today's AI excels at mathematics and chess. But Sutton highlights an often-overlooked issue:

For these tasks, the 'world rules' have already been imparted to AI by humans in advance.

But the real world is not akin to Go.

In open environments, agents must first ascertain what is worth predicting, which variables should be treated holistically, and which rules hold only in local contexts. Top athletes develop their own conceptual systems for exceedingly subtle movements; human scientists propose novel abstractions at the boundaries of existing knowledge and employ them to elucidate and plan for unknown phenomena.

Sutton contends that one of the most lacking capabilities in current AI is 'first learning a world model, then utilizing that model for planning.'

The ultimate objective of Oak Lab is to create an AI system capable of uniformly organizing knowledge, from minute perceptions to grand strategies. It should be capable of continuous training without losing control, correcting itself without descending into chaos, and absorbing new experiences while maintaining long-term knowledge coherence.

In other words, they seek not a model that merely answers questions more adeptly but a mind capable of remaining self-consistent and continuously improving over time.

This path is clearly fraught with challenges. It demands algorithms to surmount a series of unsolved problems, including catastrophic forgetting, abstraction formation, world modeling, and planning. It also means the research team must tolerate the possibility of underperforming existing products for a duration.

For large companies already entrenched in scaled revenue and existing routes, opting to 'take a step back and then switch paradigms' is no easy feat.

But this may be precisely what Sutton intends to remind the industry.

Large language models have not faltered. They have demonstrated the immense power of scaled learning and will persist as vital technological foundations. However, equating language proficiency with complete intelligence and treating training completion as learning completion may cause the industry to overlook what is truly pivotal in the next phase.

In Sutton's vision, the future will not be dominated by a single omniscient model ruling all tasks. Instead, there will be numerous 'minds' sharing the same basic design but evolving individually due to disparate experiences. They will acquire different knowledge in diverse environments, unable to exhaustively replace one another.

This may be closer to reality: The world is vast enough that no system can learn everything simultaneously. Intelligence does not entail preloading all answers into parameters but rather continuing to cultivate new understandings when encountering novel problems.

And this is precisely what Oak Lab is betting on.

By Qi