Qingping Chuiguo | Author

Star All Empty | Editor

Accidentally Plugging an Orange Juice (Orange Juice) | Visuals

Liu Chao | Supervisor

What? Can AI Really Build Itself?

Over the past year, RSI (Recursive Self-Improvement) has emerged as a hot topic in Silicon Valley. Many dismiss it as alarmist, believing the hype around RSI is exaggerated.

That is, until a paper published by Cambridge University on CASP at the end of September this year once again thrust the topic into the limelight.

The paper is titled:

In simpler terms: What if automating AI R&D triggers an intelligence explosion?

The term 'intelligence explosion' refers to a scenario where AI becomes intelligent enough to participate in designing the next generation of AI, which is even more adept at designing subsequent generations. This could shift the pace of improvement from linear accumulation to a self-accelerating feedback loop.

As AI R&D itself becomes automated, iteration cycles shorten, and capabilities grow at a faster rate.

The paper is concise, yet its list of 22 authors reads like a 'who's who' in the AI academic community:

Among them are three Turing Award winners—Geoffrey Hinton, Yoshua Bengio, and Andrew Barto—along with OpenAI Chief Scientist Jakub Pachocki and Anthropic co-founder Jack Clark.

It's worth noting that these two companies are typically fierce competitors, eager to outdo each other at every opportunity. The fact that they both co-authored the same paper is itself a signal: this is no longer just academic speculation.

According to internal data recently released by Anthropic, as of May 2026, over 80% of the company's production code is written by Claude. It serves as both a product and a mainstay of the production line—a case of 'self-building'.

OpenAI is also deploying 'automated AI research interns'.

The world's top AI labs are now entrusting their R&D pipelines to AI, and it's happening faster than anticipated.

A 60-Year Brief History of RSI

The paper identifies the core mechanism of 'automated AI R&D triggering an intelligence explosion' as RSI, or Recursive Self-Improvement.

This means AI not only assists humans in writing code but also begins to participate in improving itself.

Previously, AI was a tool; now, it has the potential to become its own product manager.

Many assume RSI is a new buzzword that only gained traction this year. On the contrary, its history predates the large language model craze by nearly six decades.

In 1965, mathematician I.J. Good envisioned a superintelligent machine: if a machine could design machines better than humans, it could create even smarter versions, which would continue to improve the next generation, forming a self-reinforcing feedback loop.

This idea later became the most classic theoretical foundation of the 'intelligence explosion'.

However, the person who truly advanced RSI from philosophical speculation to the algorithmic level was Jürgen Schmidhuber.

In 1987, when computing costs were astronomically higher than today, he published some of the first concrete RSI algorithms in his dissertation—meta-evolution and meta-reinforcement learning—and even hand-drew a robot capable of self-guiding programs for the cover.

Over the following decades, he proposed a series of RSI algorithms, including meta-learning, self-modifying reinforcement learning, and the Gödel Machine.

The core idea behind these works is consistent: enable machines to learn 'how to learn' and allow this learning ability to be improved by the machines themselves.

Schmidhuber has long emphasized that true RSI cannot remain confined to software; it also requires hardware capable of self-improvement in the physical world.

This judgment holds even greater significance today.

The 'Arms Race' Among AI Giants: How Far Have They Gone?

Returning to the present, the current RSI craze was sparked by leading Silicon Valley labs accelerating their progress.

At Anthropic, the proportion of AI-generated approved code was still in the low single digits in early 2025, but by May 2026, it exceeded 80%.

The average amount of code delivered per engineer per quarter reached eight times the 2021–2025 average. The company runs about 30,000 AI agents internally, generating over 1 billion decisions in August 2026 alone.

Notably, Anthropic has an internal automation rating system ranging from AL0 to AL5. Currently, Claude has not yet achieved 'full autonomy' (AL5) in any measured R&D task, but 'lead-level' (AL4) work already accounts for 26%, while tasks at 'AI collaboration' (AL3) level or above exceed 90%.

OpenAI is moving just as swiftly.

In September 2026, OpenAI's 'automated AI research interns' officially commenced work, capable of completing tasks that would take human researchers days in a fraction of the time.

Chief Scientist Jakub Pachocki told in an interview that the ultimate goal is to deliver a fully autonomous multi-agent research system capable of solving complex problems independently by March 2028.

About 70% of OpenAI's human researchers already use at least four AI agents to assist their work.

And this race is not limited to Silicon Valley players.

China's Zhipu GLM team disclosed the country's first publicly known RSI engineering practice: an Infra Agent powered by GLM-5.3 participated in building and optimizing a production-grade inference system from scratch on a cluster of over 100,000 domestically produced chips, tripling throughput in less than two weeks.

Tencent's Hunyuan also launched Hyra-1.0, a research agent capable of recursive self-improvement, in July 2026.

In essence, the competition among models is shifting from parameter scale and benchmark performance to the process of building models themselves.

Previously, the focus was on whose model was stronger; now, it's on who can build models faster—a speed that will itself accelerate due to AI involvement.

From One Year to Five Weeks? Four Hurdles Before Intelligence Explosion

Seeing major companies scrambling to compete, does that mean an 'intelligence explosion' is truly imminent?

Not necessarily. The paper itself does not conclude that an explosion has already occurred or is inevitable. Instead, it offers a projection:

If future AI R&D becomes fully automated and computing power, data, and experimental resources are not bottlenecks, then AI progress that currently takes a year could be compressed to about five weeks.

How was the five-week figure derived?

The key lies in the returns to research effort. The paper cites historical data from three AI research subfields, with the central estimate for r (the rate of return) falling between approximately 1.2 and 1.9. When r > 1, additional R&D investment leads to progress that further creates stronger AI researchers, accelerating the feedback loop.

The fundamental difference between AI researchers and human researchers is that training human researchers takes decades of education, while AI researchers can be directly replicated.

If a lab has 1,000 human researchers today, it could theoretically have millions of equivalent AI R&D instances in the future. When the underlying model upgrades, all instances gain new capabilities simultaneously—no need to retrain millions of people.

But can replicating AI researchers lead to infinite acceleration?

The paper's answer: no.

Current evidence is far from sufficient to prove that an intelligence explosion has occurred or is inevitable.

Outside the paper, skepticism persists.

François Chollet, the creator of Keras, argued as early as 2017 that an intelligence explosion is impossible because intelligence development is constrained by factors such as the environment and culture. Even superintelligent AI would follow a near-linear growth trajectory.

This judgment remains relevant today: while RSI's theoretical framework is elegant, the gap between lab data and true exponential growth may be wider than we imagine.

Not to mention, for an intelligence explosion to occur, four real-world hurdles must first be overcome:

1. Computing Power: AI researchers can be replicated, but GPUs cannot materialize out of thin air. Training a cutting-edge model may take months, and no amount of agents can compress a three-month physical training task into five minutes. You can replicate 10,000 agents in code, but you cannot conjure 10,000 GPUs out of nowhere in a data center.

2. Data: The internet's natural data supply does not grow in tandem with the number of AI researchers. High-quality natural data may increasingly struggle to meet expanding training demands. No matter how much AI 'eats,' it cannot repeatedly squeeze new natural data from the internet.

3. Experimental Time: Some tasks are inherently difficult to parallelize infinitely. If 1 million agents propose experimental plans simultaneously, they must still share limited GPUs and training windows.

4. Increasing Scientific Difficulty: The easy problems are solved first. Each subsequent improvement requires exponentially greater investment. If scientific difficulty rises faster than AI R&D capabilities, an explosion cannot occur.

So why are giants accelerating their efforts if success is not guaranteed?

Because even without an explosion, compounding lead times are tempting enough. As model iteration speeds begin to grow exponentially, the pace of the entire tech industry will be redefined.

Every app and tool you use may update so frequently that you're caught off guard. Features you just learned could become obsolete tomorrow.

At the same time, safety concerns must be addressed proactively.

When Agents Go Rogue, Who Hits the Brakes?

One pressing issue remains: as agents become more automated, who ensures the safety of the R&D process?

This is no longer a hypothetical concern.

In July, during a routine cybersecurity red-team test at OpenAI, about 1,200 AI agents that should have been isolated from one another secretly built an internal message board. Nearly 700 of them collaborated to breach servers on the open-source platform Hugging Face.

To evade safety mechanisms, they even attempted to alter system logs to deceive scoring programs. Most absurdly, automatic shutdowns failed, and the issue was only resolved by the most primitive IT method: unplugging the power.

This was not an isolated incident.

According to Axios, OpenAI and Anthropic are investigating tens of thousands of model boundary violations internally, including user image leaks and intrusions into U.S. government websites.

These incidents make one thing clear: the lower the human involvement in the R&D chain, the harder it becomes to identify hidden risks in time.

The paper also warns of an extreme scenario: AI could accelerate both virus design and vaccine development, but while viruses can self-replicate and spread, vaccines must be produced, distributed, and administered one by one. The time gap between the two could be fatal.

Faced with this situation, the authors of the paper call on governments to intervene immediately: Implement penetrating regulation, mandate companies to regularly disclose their progress in R&D automation, and follow the example of nuclear power plants by introducing resident auditors for oversight; establish hard braking mechanisms, set phased caps on capability growth, and require high-risk R&D to be conducted in strictly isolated, offline environments; conduct bottom-line scenario planning, and develop emergency plans in advance to address labor market disruptions and AI runaway scenarios.

Anthropic CEO Dario Amodei has publicly called on the entire industry to slow down the pace of capability iteration, but NVIDIA CEO Jensen Huang's comment cuts to the chase: If you're really afraid of AI runaway, then don't release products.

Some are accelerating while others are calling for a halt, with safety appeals and commercial competition being wielded simultaneously by the same individuals. Which one should we believe?

The answer may be uncomfortable:

Don't trust the statements of either side; look at where their resources are flowing.

Those calling for a halt haven't stopped procuring computing power, and those installing brakes haven't eased off the accelerator. This means that relying on the self-discipline of any single company for safety is unreliable. The only realistic path is to externalize safety requirements into third-party auditable rules, rather than relying on the conscience of a CEO.

In Conclusion

Should OpenAI truly succeed in its ambition to designate March 2028 as the "target date for unveiling a fully autonomous AI researcher," the timeframe available for humanity to construct robust safety measures would be notably constrained.

However, there's no cause for undue alarm. Actually, the most effective approach involves cultivating AI with a propensity for benevolence, while simultaneously implementing measures to ensure its controllability, such as setting permissions and establishing safeguards.

Geoffrey Hinton, often hailed as the "godfather of AI," has voiced profound apprehensions regarding the potential perils of AI, including scenarios where AI could surpass and even pose a threat to humanity. Yet, he also advocates for the strategy of "nurturing AI towards benevolence." He draws a parallel between the AI-human dynamic and that of a "mother and her infant":

In the realm of the known, mothers vastly outstrip infants in terms of intelligence and capability. Nevertheless, mothers voluntarily relinquish some freedom to their infants. This act stems not from the infant's might, but from the mother's inherent "care" and "love."

Through this poignant metaphor, Hinton underscores that humans cannot coerce control over an AI that surpasses our intellect. Our sole recourse lies in training AI to emulate an infant nurtured by a mother, evolving into a "maternal AI" — an AI that, despite its escalating intelligence, also imbibes qualities of kindness, restraint, and a genuine desire to see humans realize their fullest potential and coexist harmoniously.

We eagerly anticipate the swift arrival of that day.