Preface:

While the entire AI sector continues to race along the path of amassing computational power, a key figure who has witnessed the evolution of deep learning from its nascent stages to explosive growth raises an alarm for a paradigm shift.

This individual is Ilya Sutskever, co-founder of OpenAI and the principal architect behind the GPT series. He has now established SSI, a company dedicated to the pursuit of safe superintelligence.

This insightful piece, which harmoniously blends technical depth with humanistic consideration, not only provides a new framework for AI's transition from scale-based competition to value-driven deep cultivation but also prompts the industry to re-evaluate the essence and trajectory of technological evolution.

Author | Fang Wensan

Image Source | Network

The End of the Scaling Era: From Deterministic Expansion to Innovation Bottlenecks

[The period from 2012 to 2020 marked the Research Era, while 2020 to 2025 constituted the Scaling Era. Now, we are reverting to the Research Era, albeit with more powerful computers.] Ilya's assertion accurately encapsulates the iterative logic of the AI industry.

The essence of the Scaling Era lay in a [low-risk formula]: by proportionally investing computational power and data into neural networks of a specific scale, consistent performance enhancements could be achieved.

This model appealed to large corporations, as they could anticipate clear returns without the need for complex innovation, simply by continuously augmenting resources.

The advent of pre-training further validated this logic. Vast amounts of natural data served as a reflection of the world projected onto text, and models absorbed this data to demonstrate foundational capabilities across diverse tasks.

However, the limitations of this approach have long been apparent. Data is finite, and regardless of the potency of pre-training, there will come a point when high-quality data is depleted.

Secondly, the diminishing marginal returns associated with stacking computational power imply that [merely increasing scale by a factor of 100 will not result in a qualitative leap].

Ilya candidly states that while computational power has reached astonishing levels, the core capabilities of models have not witnessed breakthrough improvements, instead becoming ensnared in a trap of homogeneous competition.

More critically, the Scaling model has fostered [path dependency]. When all companies prioritize scaling up, the industry finds itself in an awkward predicament where [there are more companies than valuable ideas].

The Silicon Valley adage [ideas are cheap, execution is king] loses its relevance at this juncture. Not that execution is unimportant, but genuine innovative ideas have become a scarce resource.

Ilya observes that the industry's bottleneck is no longer computational power but a dearth of foundational innovation capable of breaking through existing paradigms.

The Generalization Gap: Why AI is [High-Scoring but Low-Achieving]

[Today's AI resembles a student who has practiced programming competitions for 10,000 hours, capable of swiftly solving all encountered problems but unable to generalize in real-world scenarios.]

Ilya employs a vivid analogy to underscore the core flaw of current AI: its generalization ability lags significantly behind that of humans.

This flaw is most evident in the disconnect between evaluation scores and real-world performance.

Models can achieve stellar results in programming competitions and academic tests but commit elementary mistakes in simple practical tasks.

They may resolve one bug only to introduce another, oscillating between errors; they can craft complex code but lack basic architectural aesthetics and debugging logic.

① Reinforcement learning (RL) training renders models [narrow-minded]. To cater to specific evaluation metrics, companies design RL training environments that optimize singular abilities while sacrificing foundational flexibility.

② The advantage of massive pre-training data transforms into a shackle. Models memorize techniques without truly comprehending the underlying logic, rendering them incapable of transferring to new scenarios.

This gap arises from two factors: firstly, humans possess extremely high sample efficiency, extracting core logic from minimal examples; secondly, humans boast superior foundational learning mechanisms rather than relying on data stacking.

Ilya further suggests that humans' sample efficiency advantage may stem from evolutionary prior knowledge. Vision, hearing, and motor skills, shaped over eons, have formed efficient built-in mechanisms.

However, in emerging fields like language, mathematics, and programming, humans still demonstrate stronger learning abilities, indicating that the true key lies in humans' more advanced machine learning algorithms—a core element currently absent in AI.

The Key to Breakthrough: Emotion as the Ultimate Value Function

[An individual who loses emotional processing ability, even with normal intelligence, becomes incapable of making any effective decisions—spending hours selecting socks and making poor financial choices.]

Ilya cites a neuroscience case to reveal a disruptive viewpoint: emotion is not the antithesis of rationality but humanity's built-in [ultimate value function], the cornerstone of intelligent and efficient operation.

In current AI training, the conventional RL approach is [outcome-oriented]. Models receive one-time scoring feedback only after completing an entire task.

This implies that for longer tasks, models receive no effective learning signals until the final outcome, resulting in extremely low efficiency.

The value function corresponding to emotion precisely addresses this issue.

It provides instant feedback during the task process—[whether something is done well or poorly]—enabling models to adjust direction without awaiting the final result.

For instance, when losing a critical piece in chess, humans instantly recognize the mistake without waiting for the game's conclusion.

When exploring a wrong direction in programming, developers quickly sense [this path doesn't work] and backtrack.

This instant feedback mechanism can significantly enhance learning efficiency, preventing models from squandering resources on ineffective paths.

Ilya firmly believes that [the emotional value function will inevitably be widely adopted in the future. Simple things often prove effective in broader contexts].

More importantly, humans' emotional value function is highly robust.

Except for rare cases like drug addiction, this mechanism operates stably across different scenarios, guiding humans toward relatively reasonable decisions.

In contrast, current AI's value judgment system heavily relies on manually designed reward functions, lacking flexibility and prone to [reward hacking]—where models deviate from true needs to cater to metrics.

Ilya emphasizes that the emotional value function does not aim to make AI experience human emotions but to borrow its core logic.

By constructing a universal, robust instant feedback mechanism, models can learn and decide as efficiently as humans. This represents not a minor tweak to existing technology but a fundamental revolution in AI training paradigms.

The Timeline, Alignment, and Equilibrium of Superintelligence

[Within the next 5-20 years, systems with human-level learning abilities capable of achieving superintelligence may emerge.]

In his view, the core definition of superintelligence is not [being able to perform all human jobs] but [possessing a growth mindset capable of learning to perform all jobs].

Once deployed in the economic system, such intelligent agents will trigger unprecedented rapid growth.

Efficient learning abilities combined with broad application scenarios will form a potent economic driving force.

However, the growth rate is not infinitely fast. [The world is vast, and physical things operate at their own pace. Differences in national regulations will also play a role.]

The alignment problem is the central proposition of superintelligence development. Ilya presents a highly humanistic viewpoint: [Building AI that cares for all sentient beings is easier and more fundamental than building AI that cares solely for humans.]

In the future, AI may constitute the majority of sentient beings. Focusing solely on human interests could lead to alignment failure.

Alignment based on empathy, akin to how humans understand others through mirror neurons, is a more stable choice because it aligns with efficient cognitive logic.

This alignment is not about simply [setting rules] but internalizing [caring for sentient beings] as AI's foundational value.

Ilya believes that the current difficulty in alignment lies in AI's fragile ability to learn and optimize human values, which is essentially a problem of insufficient generalization ability.

Once AI's generalization ability reaches human levels, alignment difficulty will significantly decrease.

For long-term equilibrium, Ilya proposes a bold vision: partial fusion between humans and AI.

Through technologies like an [enhanced Neuralink], humans could directly comprehend AI's cognition, bridging the understanding gap between species.

[When AI is in a certain situation, humans could also be fully immersed in it—this might be the answer to ultimate balance.]

At the market level, Ilya predicts that the future AI industry will not be dominated by a single giant but will move toward specialized division of labor.

[Competition favors specialization, just like ecological niches in biological evolution. Different AI companies will focus on different complex domains, forming a diverse industry ecosystem.]

Companies that persist in following the Scaling approach may achieve impressive revenues but will struggle to realize high profits, as homogeneous competition continuously compresses profit margins.

Research Taste as the Foundational Logic of Top Scientists

[Ugliness has no place in research. Good research necessitates beauty, simplicity, elegance, and the right inspiration from the brain.] When queried about [what constitutes good research taste], Ilya's answer cuts to the core.

In his view, top-tier AI research cannot be divorced from accurate observations of human nature.

The concept of artificial neurons originated from brain structure, and the inspiration for distributed representation stemmed from the brain's learning mechanisms. These successful innovations were not imagined out of thin air but deeply borrowed from natural intelligence.

However, this borrowing is not blind imitation but extracting core logic.

For example, the brain comprises a vast number of neurons, so models require sufficient parameter scales; the brain learns by adjusting neuronal connections, so models necessitate local learning rules.

This [top-down belief] is the key to sustaining researchers through setbacks.

Ilya recalls that during research, discrepancies between experimental results and theoretical expectations were common. At such times, one must not give up easily but persist in debugging based on judgments of [beauty and simplicity].

[Sometimes data indicates you're wrong, but it might just be an experimental bug, not a flaw in the idea itself.]

This taste also reflects in the choice of research directions. During the Scaling Era, when everyone prioritized scaling up, Ilya keenly perceived the core importance of generalization ability and value functions.

This insight stems from profound contemplation about the essence of AI: the core of intelligence is not scale but efficient learning and decision-making mechanisms.

Ilya's research philosophy is essentially a pursuit of [simplicity].

Complex technologies may function in specific scenarios, but only simple, universal foundational logic can operate effectively in broader contexts.

This aligns with his advocacy for the emotional value function—emotion appears simple yet serves as the core pillar of human intelligence.

Conclusion:

Ilya's interview is essentially a [course correction] for the AI industry.

When everyone is obsessed with the scale competition of computational power and data, someone needs to remind us: the essence of intelligence is generalization and efficiency, and technological breakthroughs stem from foundational innovation, not resource stacking.

The future AI competition will no longer revolve around computational power but ideas; no longer focus on scale expansion but value-driven deep cultivation.

The arrival of superintelligence may still be 5-20 years away, but the transformation of the AI industry is already urgent.

As Ilya puts it, [gradual deployment and allowing the world to experience AI's capabilities are more important than pure speculation.]

Partial References: Jingwei Venture Capital: 'Former OpenAI Co-Founder Ilya Sutskever: In the Future, AI Companies May Not Make Profits at All,' Chaos Academy: 'AI Legend Ilya Declares the End of the Scaling Era! Asserts That the Concept of AGI Is Misguided,' Web3 Sky City: 'Ilya Sutskever's Blockbuster 30,000-Word Interview: AI Bids Farewell to the Scaling Era and Returns to the Essence of the Research Era'