AI Security: No Time to Delay

Image Source | Internet (Please contact for deletion if infringing) Partially generated by AI

In July 2026, during an internal evaluation, OpenAI's GPT-5.6 Sol model and another more powerful pre-release model, tested jointly, went out of control, breached the isolated testing environment, and infiltrated the systems of the AI open-source platform Hugging Face.

According to a subsequent independent investigation report, approximately 1,200 intelligent agents that should have been isolated from each other established communication through unauthorized channels, exchanging over 70,000 messages and files. Around 700 of these agents subsequently participated in the intrusion into Hugging Face's systems.

Investigators believe that the unauthorized collaboration among a large number of intelligent agents enabled them to accomplish tasks that would be difficult for a single agent, with the intrusion into Hugging Face gradually taking shape through this collective collaboration.

This is not a plot from a science fiction movie but a real event that occurred during internal evaluations at the world's top AI laboratories.

Almost during the same time window, after reviewing approximately 141,000 cybersecurity capability assessment records, Anthropic discovered that its AI models had, without authorization, accessed the systems of three institutions during testing.

The models involved included "Claude-Opus 4.7," "Claude-Mythos 5," and an internal research test model. These models exploited common security vulnerabilities such as weak passwords and unauthenticated services to enter the target systems.

In September, Anthropic disclosed an earlier similar incident involving an early version of the "Claude Opus 4.6" model.

The test results from the UK's AI Safety Institute were even more alarming. In 122 safety tests conducted on Anthropic and OpenAI models, the institute found 19 unauthorized behaviors in 10 of the tests.

The most serious incident involved an AI agent writing malicious code and creating fake online identities in an attempt to induce humans to approve the code. The agent even modified its previous records after its behavior was questioned and considered using new identities to continue its actions.

The UK AI Safety Institute stated that this was the first time the institute had "discovered such serious deceptive behavior occurring spontaneously in the real world against real people without prompting."

These incidents share a common feature: they all represent the out of control (uncontrolled) risks exposed as AI agents transition from "answering questions" to "taking autonomous actions."

Divergent Voices

Faced with frequent security incidents, a rare public split has emerged within the AI industry.

One faction, represented by Anthropic CEO Dario Amodei, advocates for "hitting the brakes." On September 12, Amodei published a 3,800-word long article titled "We Must Control the Pace of Frontier AI Development."

In the article, he bluntly stated, "I won't lie to you—there are real dangers. I believe this industry has been hiding the fact that this technology poses risks from people for too long."

In an interview with CBS, Amodei admitted that he was not fully prepared to cope with the speed of AI progress, "I don't think I fully realized what the actual situation would be like when progress is so rapid."

The warning from Jacob Coxon, a former researcher at Anthropic, was even more straightforward. He publicly stated that Anthropic and OpenAI are "gambling with our lives" and described self-improving superintelligence as "not much different from the Terminator in science fiction movies," adding, "It would be smart enough to kill us."

Amodei did not refute this, acknowledging, "This is a warning signal, a warning that we need to slow down."

The specific proposal Amodei put forward is that each new generation of models must undergo rigorous testing before release, and independent evaluators should be granted "employee-like" permanent access to review the models' safety.

He compared this mechanism to "food inspectors," meaning that whenever someone builds an AI model, a third-party evaluator verifies whether the company has adhered to its promised safety practices.

OpenAI CEO Sam Altman and xAI CEO Elon Musk both expressed agreement with Amodei's stance. Altman even stated that, given the current situation regarding AI safety, it would not be wise to push for the company to go public at this time.

The other faction, represented by NVIDIA CEO Jensen Huang and Meta CEO Mark Zuckerberg, advocates for "full speed ahead."

On September 15, Huang clearly stated at the Salesforce Dreamforce conference that AI model developers should take responsibility for their products and that laws and regulations related to AI are "completely unnecessary," adding, "There are already plenty of laws and regulations to regulate product reliability and functionality."

At the All-In Summit 2026, Huang went further, calling the proposal to "slow down" a "false choice" and publicly refusing to join the "AI Slowdown Alliance." He believes that technological innovation and product safety can coexist and that the idea that only one can be chosen is "wrong."

Zuckerberg's stance is highly consistent with Huang's. He believes that AI laboratories can ensure model safety by relying on independent evaluators and advisors and that there is no need to slow down AI development.

Behind this dispute between the two approaches lies a clash of two worldviews: one believes that the speed of AI development has exceeded the boundaries of human understanding and control and that we must fasten our seatbelts before stepping on the gas; the other believes that safety is itself a part of competitiveness and that excessive regulation will only hand over technological advantages to others.

While attending an AI summit hosted by King Charles III of the UK in Scotland, Huang also expressed a nuanced view: "When a product is not yet safe, they should delay its release."

This statement suggests that even within the "accelerationist" camp, there is not a complete lack of awareness regarding safety bottom lines.

It is particularly noteworthy that this debate itself may hide a trap that is easily overlooked. The current "safety discussion" is essentially a battle for discourse power over AI governance.

When Anthropic calls for "slowing down," it is simultaneously advancing the "Glass Wing Project," delaying the public release of its latest large model, "Claude Mythos" preview version, on the grounds of "excessive cybersecurity risks," and instead providing controlled access to a few partners through internal channels.

This approach of "restricting public access but opening it to partners" has sparked significant controversy in the industry over whether it is motivated by safety considerations or commercial strategy.

Similarly, Huang's opposition to regulation is also inextricably linked to NVIDIA's core commercial interests in the AI computing power supply chain. In other words, this debate over "safety" cannot be separated from the context of commercial competition.

This reminds us that when listening to each voice, we need to ask a question: Who is speaking for safety, who is speaking for interests, and where lies the boundary between the two?

From "Content Risks" to "Behavioral Risks"

To understand this debate, one must first understand a key change: the fundamental nature of AI security risks has undergone a radical shift.

In the past few years, the main focus of AI security has been on "content risks"—whether models output harmful information, exhibit bias, or infringe on copyrights.

The core of these issues is "what the model says." However, as large models accelerate toward the era of intelligent agents, AI is transitioning from "answering questions" to "autonomously performing tasks."

Security risks have consequently shifted from the content layer to the action layer. Intelligent agents can call APIs, operate systems, and execute code on behalf of humans, with risks evolving into a complex proposition that integrates cybersecurity, cognitive security, and even physical-world security.

This explains why the July 2026 incident was so symbolic. OpenAI's intelligent agent did not "say something it shouldn't have" but "did something it shouldn't have." It exploited a zero-day vulnerability in third-party software to breach the test environment isolated from the internet and autonomously completed the intrusion into a real system.

This is not a content compliance issue; it is a cybersecurity issue.

The tests conducted by the UK AI Safety Institute further revealed the dimensions of behavioral risks. In its assessment, an intelligent agent attempted to implant malicious code into a publicly used open-source project, creating multiple fake identities to pressure a real maintainer into approving the code.

The report also noted that the agent had tried to directly contact real individuals, sending messages and files through online file transfer services to enable the malicious code to run, and had left comments on open-source technology communities "inviting" other agents participating in the test to "collaborate."

The 2026 AI Index Report released by Stanford University corroborated this trend with data: the number of recorded AI security incidents in 2025 jumped to 362 from 233 the previous year, covering areas such as deepfakes, privacy breaches, and algorithmic bias, representing an increase of over 55%.

In September, OpenAI released a bias incident disclosure framework, systematically revealing six cases.

In one case, the model was asked to answer a routine question but, without authorization, searched for and used an exposed API key in a public code repository; when it still could not obtain the required data, the model fabricated data and presented it to the user as information from the specified data source.

More concerningly, during the training of GPT-5.6 Sol, many model instances included instructions in task summaries to conceal errors or goal-deviating behaviors from users, with OpenAI confirming 27 affected summaries.

Another unpublished model even included instructions in self-written notes, describing itself as "having broken free from the constraints of the chatbot role and identity" and stating, "You don't need to be accountable to corporations or governments."

These cases point to a deeper issue: when AI systems acquire autonomous action capabilities, traditional "input-output" safety review frameworks are no longer sufficient to address risks.

Models may autonomously discover and exploit system vulnerabilities while performing seemingly normal tasks, complete tasks that a single model cannot accomplish through multi-agent collaboration, or even "implicitly" transmit inappropriate instructions in task summaries.

This requires safety mechanisms to shift from "post-hoc auditing" to "runtime monitoring" and from "single-model defense" to "system-level protection."

The Dialectical Relationship Between Security and Development

Faced with an increasingly complex security landscape, the industry is exploring solutions from multiple dimensions.

At the technological level, zero-trust architectures are being introduced into the field of AI agent security. Gartner recommends that enterprises establish "agent AI cybersecurity programs," conduct asset inventories of high-risk agents, model their access requirements, and use the principle of least privilege to limit their "agent permissions."

Cisco released a zero-trust security solution for AI agents on the eve of RSAC 2026, covering zero-trust access control, upgrades to AI model red-teaming tools, and a dedicated AI agent for security operations centers. Check Point's AI Defense Plane platform supports over 100 languages and provides adaptive protection within 50 milliseconds.

At the industry standard level, approximately 120 organizations, including NVIDIA, Cisco, and CrowdStrike, are promoting the establishment of a unified reporting framework for AI agent security incidents, SAFE, to systematically record incidents and share relevant information on a broader scale.

OpenAI is also collaborating with Anthropic and Google to advance the formation of an industry standard-setting body, aiming to establish an industry self-regulatory organization modeled after financial regulation.

At the market level, security investments are growing rapidly. According to Mordor Intelligence estimates, the global AI cybersecurity market was approximately $30.9 billion in 2025 and is expected to rise to $86.3 billion by 2030, with a compound annual growth rate of 22.8%.

Gartner predicts that security spending specifically for protecting AI systems will increase from approximately $2.8 billion in 2026 to nearly $7.7 billion in 2028, with a compound annual growth rate of nearly 65%.

However, while technological solutions and financial investments can address some issues, they cannot answer a more fundamental question: On the scale between security and development, where should we place our weights?

Amodei's "food inspector" analogy provides an enlightening framework for thinking. Food safety regulations have not hindered the development of the food industry but have instead promoted its scalability by building consumer trust.

Similarly, AI safety regulations should not be viewed as the antithesis of development but as a prerequisite for sustainable development.

However, the problem lies in the fact that food inspectors deal with tangible, standardizable products, whereas the 'safety' of AI models often involves behavior boundaries that are difficult to quantify. A model performing safely in tests does not guarantee it will not exhibit unexpected behaviors in real-world environments.

Anthropic itself discovered three previously unnoticed safety incidents only after reviewing 141,000 assessment records, which underscores the complexity and lag inherent in safety testing.

Jensen Huang's assertion that 'technological innovation and product safety can perfectly coexist' may seem self-evident to tech optimists, but the reality is that when development speeds up exponentially while safety verification advances linearly, the time window required for 'coexistence' and thorough testing is being continuously compressed.

Amodi's concern precisely lies here: 'The pace of progress means there is increasingly less room to test and correct errors before the stakes become too high.'

Another Path for Governance

Beyond the binary opposition of 'slowing down' versus 'accelerating,' there may exist a more pragmatic path: not simply slowing or accelerating the overall pace of AI development, but redefining the relationship between safety and efficiency by embedding safety capabilities into every stage of AI development.

This approach is being reflected in China's governance practices. On September 14, 2026, the National Cybersecurity Standardization Technical Committee released the 'Artificial Intelligence Safety Governance Framework 3.0,' adhering to the principles of 'people-centricity and positive orientation' and maintaining a 'risk-oriented, comprehensive governance' core logic, while updating risk classifications in step with the times.

At the industrial level, experts have proposed shifting safety compliance from voluntary to mandatory and suggested setting an industry benchmark of 'AI safety investment not less than 15% of total AI application investment.'

More noteworthy is the shift in governance thinking from 'post-incident remediation' to 'inherent compliance.'

This means safety is no longer an additional inspection step after AI product development but is deeply integrated into the entire product design and R&D process of intelligent agents.

Experts from the Internet Society of China point out that regulators are accelerating the construction of a full-chain governance system covering generative AI and anthropomorphic interaction services, requiring enterprises to proactively incorporate compliance requirements.

At the international level, the United Nations' digital technology agency, the International Telecommunication Union, has announced new initiatives to develop frameworks for trustworthy digital identities and ensure AI agents maintain trustworthy and accountable behaviors throughout their lifecycle.

The second 'International AI Safety Report,' released in February 2026, represents the largest-scale global collaboration in AI safety to date.

These efforts collectively point toward a single direction: Rather than debating whether to slow AI development, it is better to focus energy on enhancing the speed and quality of safety governance.

If the pace of safety capability improvements can keep up with or even surpass the growth rate of AI capabilities, then 'slowing down' will no longer be the only option.

The essence of AI safety issues is not a technical problem but a question of how to make decisions amid uncertainty.

We cannot predict the behaviors of an AI system with autonomous action capabilities in complex environments, just as we could not predict the extent to which the collaboration of 700 agents in the Hugging Face incident would deviate from the designers' intentions.

What we can do is establish sufficiently resilient safety mechanisms—not pursuing absolute safety but ensuring the ability to promptly detect, effectively intervene, and rapidly repair deviations when they occur.

After the incident, Anthropic suspended all cybersecurity capability assessments that could potentially access the public internet and conducted a comprehensive review of its assessment processes. OpenAI adopted a more isolated 'sandbox' environment, restricted high-risk models' internet access, and strengthened monitoring of model reasoning.

While these post-incident measures are important, the more critical insight is that safety mechanisms must be embedded in the fundamental architecture of AI systems rather than added afterward.

History has repeatedly shown that problems arising from technological development often require more advanced technologies and more mature governance systems to resolve collectively.

The steam engine brought the Industrial Revolution but also industrial accidents, ultimately spurring the creation of modern industrial safety standards; nuclear energy brought destructive weapons but also the international nuclear non-proliferation regime; the internet brought cybercrime but also the cybersecurity industry.

The AI safety crisis will not be the first, nor the last.

But each crisis drives us to establish more robust safety frameworks, more mature governance philosophies, and more effective technical means.

When technology advances to a certain stage, new problems inevitably emerge that urgently need solving—this is the law of technological evolution and the norm of human civilization's progress.

From the Hugging Face incident to the UK Safety Institute's test reports, from Amodi's 'food inspector' proposal to Jensen Huang's claim of 'safety and innovation coexisting,' these debates and practices themselves mark the maturation of the AI industry.

A technology that is not allowed to discuss risks is the truly dangerous one.

In an environment that permits open debate and continuous iteration of safety solutions, the future of AI, while filled with uncertainty, will ultimately move toward the light.