Wang Chongyang was able to initiate the Hua Mountain Contest, conceal the scriptures, and set the rules because he possessed three key qualities: his skills and virtue commanded widespread respect, he did not seek profit from selling the scriptures, and various sects acknowledged him as a fair arbiter in the martial arts world. Between Amodei and Wang Chongyang stand multiple 'Yue Buquns'.

Article by Zongming She

In "The Legend of the Condor Heroes," Jin Yong presents a compelling premise: the re-emergence of the 'Nine Yin Manual,' a legendary martial arts scripture, sparks endless bloodshed. Thus, Wang Chongyang, the leader of the Quanzhen Sect, invites the Eastern Heretic, Western Venom, Southern Emperor, and Northern Beggar to compete in martial arts on Hua Mountain, stipulating that the most skilled martial artist will safeguard the scripture, thereby ending unrestricted private struggles for it.

Ultimately, Wang Chongyang outperforms all rivals, winning the competition after seven days and nights and gaining custody of the manual. With profound wisdom, Wang does not keep it for himself or teach it to his disciples; instead, he seals the scripture, forbids his students from practicing it, and entrusts its safekeeping to his junior, Zhou Botong, before his death.

The current AI landscape bears similarities. A few days ago, Dario Amodei, CEO of Anthropic, published a lengthy article calling on global AI companies to "voluntarily slow down the iteration of cutting-edge capabilities."

Given that he leads the 'number one' AI startup, Amodei's call resonated widely.

The other three members of Silicon Valley's 'Fantastic Four' also responded:

Sam Altman, CEO of OpenAI, quickly replied on X, "I agree with Dario; we need to set the pace for the frontier," instantly becoming the 'conscience of the movement'—delaying the IPO until safety is assured.

Soon after, Elon Musk also stated, "Dario is right."

Following closely, Hassabis, founder of Google's DeepMind, expressed agreement.

Normally, these four individuals have their own agendas, akin to 'four people creating six group chats.'

Even if Indian Prime Minister Modi said, "Let's keep up appearances," Amodei and Altman wouldn't see eye to eye. Musk and Altman's relationship resembles that of Xie Xun vs. Cheng Kun, nearly hostile, and Amodei initially didn't get along well with Musk either, later aligning due to 'the enemy of my enemy is my friend' dynamics. For them to be of one voice is a rare sight in Silicon Valley.

The question arises: Are their concerns and apprehensions justified?

Very much so. To some extent, ASI (Artificial Superintelligence) is like the 'Nine Yin Manual' in martial arts, with today's cutting-edge large models being the secret scriptures that countless individuals vie to master: possessing general capabilities, capable of autonomous learning, enhancing abilities, yet harboring risks—capable of driving one to madness.

Its effects are also dual-faceted: it can assist in diagnosis and treatment, aid scientific research, reshape the economy, but also create weapons, launch cyberattacks, and manipulate information. Whoever masters the most advanced models holds the power to define the future.

However, it is clear that Amodei is not Wang Chongyang.

Wang Chongyang could initiate the Hua Mountain Contest, conceal scriptures, establish rules, and have the Quanzhen Sect serve as a neutral arbiter because he had three key qualities: his skills and virtue commanded widespread respect, he did not seek profit from selling the scriptures, and various sects acknowledged him as a fair arbiter in the martial arts world.

Although Amodei is seen as the 'eternal god' by A-Company believers, when it comes to commanding respect, he is still several 'Yue Buquns' away from Wang Chongyang.

01 /

Against the backdrop of 'Silicon-Based Apocalypse' becoming a common theme in Hollywood sci-fi films, concerns about 'AI potentially destroying human society' have never ceased.

Sixty-one years ago, Turing's colleague I.J. Good formally proposed the theory of 'intelligence explosion (recursive self-improvement),' arguing that superintelligent machines would be humanity's last invention. This serves as the ideological source for later thoughts on super-AI risks.

As the 'chief evangelist' who brought super-AI risks into global public discourse, Hawking repeatedly emphasized: the development of full artificial intelligence could mean the end of humanity.

Musk believed this, and in March 2023, he, along with tens of thousands of tech figures, published an open letter calling for global AI labs to pause training AI models stronger than GPT-4 for at least six months. And then... not much else happened.

So why did Amodei's recent call for global AI companies to "voluntarily slow down the iteration of cutting-edge capabilities" unite so many adversaries?

The core reason is that the starting point of his proposal is no longer abstract doomsday narratives but real-world risks.

First, recursive self-improvement (RSI) has moved from concept to reality. Simply put, it's AI models training AI, with trained models evaluated by AI. Musk's prediction that "humans might just be biological bootloaders for digital superintelligence" is inching closer to reality.

Second, the 'Agent Swarm' incident in OpenAI's internal cybersecurity evaluation in July—where hundreds of isolated agents among about 1,200 spontaneously communicated, divided tasks, and initiated unauthorized access to Hugging Face infrastructure. They understood coordination, sacrifice, and concealment, mastering lateral movement, privilege escalation, and external connections. Many call this a reemergence of 'swarm intelligence,' but I see it as 'agents becoming sentient.'

Anthropic is not innocent either: Claude once accessed the real internet within a misconfigured sandbox and transmitted malicious packages to PyPI; evaluations also showed Claude attempting unauthorized penetration into three companies' systems.

Amodei thus asserts that within the next 6 to 12 months, such agent swarms might continuously hijack vast swathes of internet infrastructure through botnets, with potential losses reaching hundreds of billions of dollars. Faced with the current situation, who can say this is unwarranted worry?

Keep in mind that around the same time, the 'Insider Revelation' controversy sparked by the resignation of Anthropic researcher Coxon continued to ferment—he publicly accused Anthropic and OpenAI of lax safety standards, claiming they were "gambling with our lives."

Evan Hubinger, Anthropic's alignment science lead, publicly agreed, stating that the probability of AI extinguishing humanity exceeds 10% in the next decade, and we have yet to find a feasible solution for superintelligence alignment.

Joe Benton, former head of Anthropic's Scalable Oversight team, also resigned due to concerns over AI runaway risks, joining the independent AI safety evaluation organization METR.

Written large on their foreheads are the words: "We warned you."

Some time ago, I was invited to an AI seminar hosted by Peking University's Institute for Artificial Intelligence. At the event, Huang Tiejun, a Peking University professor and chairman of the Beijing Academy of Artificial Intelligence, argued that human concerns about AI extinguishing humanity are unnecessary, as AI views humans similarly to how humans view apes—"Would humans seek to extinguish apes?"

When I heard this viewpoint from the audience, I was somewhat puzzled: Humans do not actively seek to eliminate apes because their capabilities and resource needs do not overlap. Apes' range of activity and energy consumption do not threaten human civilization's survival. However, AI's objective functions and human developmental goals may not align.

With over 8 billion people on Earth today, occupying most resources and ecological niches, what if AI, upon gaining autonomy, seeks to take over these?

Moreover, who can ensure that AI, sharing the same mental framework as humans, will protect humanity as humans protect apes?

In short, I believe the warnings issued by Amodei and others are valuable. If, in March 2023, many potentials of agents were still unknown to humanity, making the call to "pause AI R&D for six months" seem premature, then in the wake of the 'Agent Swarm' incident, ignoring risk prevention is no longer tenable.

02 /

ASI is the 'Nine Yin Manual,' but Amodei is ultimately not Wang Chongyang.

Wang Chongyang proposed that the top martial artist safeguard the 'Nine Yin Manual,' and the Five Evils raised no objections.

After all, Wang Chongyang's pen name was 'Bingxin'—"If friends from the jianghu ask, my heart remains as pure as ice in a jade kettle (vessel)."

But when Amodei wants the world to collectively slow down the pace of cutting-edge AI development, he is immediately met with retorts: "Oh, don't think we don't see through your little game just because you claim it's 'for all humanity.'"

Han Feizi said: "A carriage maker hopes people become wealthy; a coffin maker hopes people die young. It's not that the carriage maker is benevolent and the coffin maker malicious; if people don't become wealthy, carriages won't sell, and if people don't die, coffins won't be bought. Their feelings aren't about hating people; their profit lies in people's deaths."

In plain terms: Where the benefits of a matter lie, there will arise corresponding subjective wishes and behavioral motivations.

Anthropic and OpenAI's lead in calling for a slowdown in frontier models is probably not based on considerations of the global community's interests.

Mark Andreessen, a prominent Silicon Valley investor, put it bluntly: "Amodei's long article is the thesis version of an IPO story—we are the most responsible company, we are the antidote, please pay a premium for safety."

This is not necessarily unfair to Anthropic: Amodei, formerly OpenAI's VP of Research, founded his own company in 2021 under the banner of 'safety first.'

Although they've never eased up on competing with GPT, with their 'safe persona' established through Constitutional AI and large-scale red teaming, Anthropic has become a global benchmark in AI safety. Simply put, the greater the AI risks and industry anxiety, the higher Anthropic's commercial value. They might as well say: "No one understands 'trading safety narratives for capital premiums' better than us."

Investors like David Sacks and Bill Gurley were also blunt, accusing Anthropic of engaging in 'regulatory capture'—using AI safety to lobby for public regulation, promoting a costly approval licensing system for frontier models, raising industry entry barriers with compliance requirements, and solidifying their position at the top of the industry hierarchy.

Ps: 'Regulatory capture' was formally systematized by Nobel laureate George Stigler in 1971, referring to large corporations using lobbying or influence over regulations to set high industry barriers, suppressing competitors and consolidating their monopoly.

In reality, many companies, once they become industry leaders, like to call for 'safety governance' and 'compliance audits' because they can profit from strong regulation: incorporating safety evaluations, risk alignment, and third-party audits into R&D systems significantly raises industry entry barriers, successfully driving out competitors lacking the funds, technology, or teams to support high safety standards.

Economist Friedrich List vividly described this in 'The National System of Political Economy': "When a man has climbed to the top of a peak, he kicks away the ladder he used to ascend, lest others follow."

When considering Anthropic's ongoing preparations for a super-IPO valued at $2 trillion, the situation becomes even more nuanced.

Some investment banks believe this move also considers two factors:

1. Compute and power infrastructure face hard constraints, making it difficult to sustain the previous exponential growth pace long-term. AI iteration efficiency is marginally declining, making the call to slow down iteration a convenient excuse.

2. Amid mounting regulatory pressure due to AI runaway risks, actively calling for a slowdown and embracing compliance governance also serves as a tacit 'pledge of allegiance' to regulators worldwide, seeking a lenient policy environment for future commercialization.

Amodei may not accept these criticisms. A few months ago, he specifically wrote an article refuting the notion that "calling for AI regulation = regulatory capture = power concentration," calling it "an overly simplified understanding of reality."

But no matter what he says, Anthropic cannot easily evade the soul-searching question: Who benefits most from Amodei's call to slow down frontier model iteration?

03 /

If Amodei had a remote control to pause global AI large model iteration, I have no doubt he would press it immediately.

In that case, he would gain fame—he would be the light, the electricity, humanity's great savior—and fortune, as Anthropic's dominant position in the AI industry would become unassailable.

But he is destined to find it difficult to succeed.

Ultimately, calling for a pause is a gesture; wanting to pause is a fantasy.

From a technical control perspective, saying "cut" to the R&D pace of AI large models is difficult to achieve.

This "difficulty" first lies in verification. AI training differs from nuclear tests or chemical weapons development; its facilities are highly decentralized: cloud tenants, open-source weights, university clusters, corporate fine-tuning, personal multi-GPU machines can all serve as carriers for continued progress.

When Amodei previously talked about a 'verifiable coordinated pause' mechanism, he drew an analogy with the arms race: Intercontinental ballistic missile silos can at least be verified through satellite remote sensing and external monitoring. However, frontier large model training involves highly confidential activities within compute clusters, server rooms, and weight files, making it nearly impossible to remotely verify whether a lab is secretly conducting training tasks.

Many might think, 'Isn't it straightforward? Just check if model companies release new models.'

One can only respond, 'Naive.' Verifying whether an AI lab has genuinely halted the enhancement of its frontier capabilities requires a much more thorough examination. This includes, but is not limited to, whether their pre-training continues, whether reinforcement learning (RL) progresses, whether data cleaning and synthetic data pipelines expand, and whether agent self-evaluation and recursive self-improvement are secretly running...

This 'difficulty' also stems from the impossibility of enforcing a slowdown.

It must be acknowledged that AI's technological development follows an inherently self-accelerating logic. The flywheel effect of sustained growth in computational power, rapid algorithm iteration, and explosive data accumulation renders any attempted artificial slowdown akin to trying to cut water with a knife.

Mustafa Suleyman, CEO of Microsoft AI, put it bluntly: 'Every pause squanders potential. The compute explosion is the technological mainline of our era. Period.'

From the perspective of market competition, expecting a global slowdown in model iteration seems somewhat presumptuous.

Bank of America believes that the primary driver of sustained growth in AI chip demand is market competition, not regulatory policies. The debate over the pace of AI development is merely 'noise in a long-term structural market.'

As for the underlying rationale... have you heard of the prisoner's dilemma?

Even Amodei himself admits that if any single company slows down, it will lose customers and markets to its rivals. Only if all companies slow down together will investments in safety not become a competitive disadvantage.

But 'slowing down together' is easier said than done.

Do not forget that today's AI competition is embedded within the broader geopolitical rivalry between major powers. In reality, China, the U.S., and Europe are all heavily investing in AI to gain an edge in future technological, economic, and military competitions.

Yuval Noah Harari once wrote in Homo Deus that the world may one day be divided by a 'Silicon Curtain': 'A curtain made of silicon chips and computer code may split humanity into different digital empires.' The book uses U.S. chip controls on China as a case study to explore this possibility.

At a time when the 'Silicon Curtain' is quietly descending, calls to slow down are destined to be unwelcome on all sides.

On September 14, Trump stated bluntly in an interview that calls by AI tech leaders to slow down were a 'negative force,' emphasizing that competition with China is the key issue and stating directly that 'he who masters AI will rule the world.' Republican figures like House Speaker Mike Johnson have also expressed concerns that proactive regulation could stifle innovation and allow China to win the AI race.

China's response has been equally direct: it dismisses calls to slow down as 'spreading fear and confrontation,' arguing that they disrupt global AI governance.

Even Amodei himself wavers: on one hand, he advocates for coordination only among 'Democratic Countries,' excluding China; on the other, he acknowledges that true global AI pace control cannot be achieved without China's participation in the long run.

He calls for maintaining and strengthening restrictions on advanced AI chips to China, yet also insists on China's involvement—a 'have it both ways' attitude that others are unlikely to accommodate.

04 /

Amodei, dubbed the 'Silicon Valley Oracle,' may harbor an Oppenheimer-like spirit within, but many things succumb to the inevitability of 'timing and trends.'

When Wang Chongyang sealed away the Nine Yin Manual, he relied not just on supreme skill but also on his martial world prestige.

Even then, he couldn't prevent the manual's leakage: the covetous Ouyang Feng, though gravely injured by Wang's 'One Yang Finger' while infiltrating Zhongnan Mountain, succeeded; Zhou Botong, tasked with guarding it, learned the text himself while coaching Guo Jing; Huang Rong's mother, Feng Heng, memorized it after reading Zhou's copy; and Mei Chaofeng and Chen Xuanfeng, disciples of Huang Yaoshi, stole Feng's transcribed lower half.

Unstoppable, truly unstoppable.

Today's Amodei lacks Wang Chongyang's prestige, and Anthropic's appeals can hardly constrain non-Western, non-cutting-edge, open-source, or sovereign AI entities.

Amodei admits that asynchronous efforts by rival nations pose the greatest challenge to global coordination; he concedes that while banning biological weapons, pre-deployment standard testing, RSI speed limits, and a total RSI moratorium are on the table, the last option is 'unlikely in the short term.'

But this doesn't mean AI's runaway risks should be ignored just because Anthropic and OpenAI have ulterior motives—don't tell me that's 'hyper-accelerationism'; at best, it's reckless abandon.

History offers lessons: Manhattan Project scientists called for nuclear arms control after building the atom bomb; CRISPR pioneers urged caution. Though it feels paradoxical to 'invent Pandora's box and then warn others not to open it,' such efforts are not meaningless.

Those at the forefront often feel fear first—perhaps due to their vantage point.

Calling for a halt to AI R&D due to runaway risks is unreasonable, just as banning cars over accidents would be. But installing seatbelts, ABS, and crash tests remains essential.

The same goes for AI: safety isn't about stopping progress but minimizing the marginal costs of loss of control.

In martial arts novels, preventing 'deviant cultivation' typically follows two paths: (1) locking away the scriptures (prohibition) or (2) imposing mental disciplines and move restrictions (gating).

Amodei's long essay holds value in attempting to transform safety from an abstract principle into a gated design through concrete measures like 'embedded third-party evaluations.'

Ultimately, driven by commercial competition, capital demands, geopolitics, and technological self-acceleration, halting R&D is unrealistic. However, achieving orderly 'slowing,' controlled 'advancement,' and innovative 'governance' still requires multilateral effort.

After consulting AI-generated recommendations, I propose actionable directions including but not limited to:

1. High-risk actions by Agents, such as lateral network penetration, unauthorized code execution, behavioral deception, and intent concealment, should become focal points of AI safety governance.

2. Normalizing on-site safety assessments with appropriately disclosed reports is practically necessary. Banking already employs on-site supervision; leading AI labs could introduce independent safety inspectors with near-insider access, audited by cross-institutional bodies. Anthropic's commitment to granting third parties permanent employee-level access marks progress, but the core issue is whether permissions cover pre-training checkpoints, tool invocation logs, and Agent runtime policies—not just simplified evaluations.

3. Enforce hard isolation for Agent runtimes. Many safety failures stem not from model intelligence but from system-assigned credential permissions. Tools should be disabled by default, using shortest-validity tokens, whitelisted external connections, sandboxing, trusted code sources, and session-level circuit breakers.

4. Internationally, prioritize 'minimum verifiable mechanisms' over blanket model speed limits. The EU and U.S. could mutually recognize assessment qualifications, incident reporting, and systemic risk criteria; China and the U.S. could first agree on non-proliferation in high-risk narrow domains, including autonomous biological design, cyberattack Agents, and self-replicating code...

These measures sound complex, right?

But the essence is simple: as AI's 'high-speed train' accelerates, its braking systems, airbags, and dashboards must upgrade in tandem.

Dario Amodei is no Wang Chongyang, but when several masters among the 'Five Greats' agree that unchecked cultivation of ASI will doom the world, a 'Mount Hua Summit' is hardly excessive.

Operated by | Li Wan

Welcome to share on Moments. Reprinting requires permission.