Original: Shenmou Finance (chutou0325)

In recent years, AI has been steadily weaving itself into the fabric of daily life. At the office, AI-powered tools can help you draft weekly reports and organize meeting minutes; after work, an AI ride-hailing assistant gets you a car home with a single tap; once you're back and browsing, AI-generated short dramas fill the screen to keep you entertained. Today's large AI models are no longer confined to being chat companions in a dialog box—they have already become "quasi-assistants" capable of solving problems at the desk, on the commute, and on the screen.

AI is advancing faster and faster, yet a fatal weakness has always capped its ceiling: most of its abilities are "frozen." A plan revised countless times today must be started from scratch tomorrow in a new conversation; the same pitfalls it stumbled into this time will trip it up again next time. Its capabilities are frozen at the moment it leaves the factory.

If it forever spins in the cycle of factory and scrap, AI cannot achieve a true leap. For this reason, AI researchers have set their sights on a more fundamental breakthrough: letting AI "research and develop itself." Academia calls this recursive self-improvement, or RSI for short.

To make RSI truly turn, the first thing to do is not "become stronger," but to first remember "what can make itself stronger." This involves a core module of AI: "memory."

Every effort people make to advance AI memory mechanisms solves a problem, while also taking humans further from directly understanding it.

01 The "Static Answers" That Can Be Controlled

In the past, when talking about AI memory, the industry usually had two approaches. One is an external data repository, retrieved when needed; the other is adjusting model parameters so that memory "grows in." The former can be looked up, but cannot change the next default action; the latter can be changed, but is difficult to review line by line. Although each has its own characteristics, they share a common fault line: memory is stored, but what to remember and what not to remember has always been pre-screened by humans, and AI is only responsible for reading.

But once AI is to accumulate experience on its own during operation, this premise no longer holds. What it faces is no longer whether memory exists at retrieval time or whether parameters have been changed, but whether the thing it records is itself true, correct, and conducive to self-improvement.

This is also the inherent flaw of this memory mechanism: it can only store, not judge.

A paper on memory architecture last December provided an intuitive comparison. On the same 20B open-source model, if only full-context memory is relied upon, the accuracy on the LongMemEval long-range memory benchmark is only 39.0%; after introducing a structured external memory architecture, the accuracy can rise to 83.6%. So the gap is not in "whether it was stored," but in "whether what was stored has been organized and verified."

Previously this step was done by humans: humans screened materials, humans tuned parameters, humans introduced mechanisms, and the memorized content had all been tested. Once AI records by itself, it may mistake erroneous memories for correct ones, or treat a small-probability event that happened to succeed by chance as universal truth.

In July this year, an RSI review covering 1,250 papers pointed out that merely letting AI store memories is not enough; what is stored must also be verified and trustworthy. If what is recorded is itself wrong, the system will slide toward self-confirmation loops, model collapse, diversity collapse...

Simply put, AI records something it has no ability to judge as right or wrong, and then uses it as the basis for the next step, using a set of erroneous logic it generated itself to verify another erroneous conclusion it generated itself, and finally drifts further and further astray within this logical closed loop. In the absence of external intervention, such errors may be continuously amplified, ultimately destroying the entire system's judgment like a source of contamination.

Therefore the key question is not whether to let AI record by itself, but who judges right and wrong on its behalf.

To solve this problem, the power of judgment must be taken out of AI's hands and given to an external verification mechanism. Academia calls this solution "pre-submission gating." This mechanism does not participate in content generation; it only recognizes objective code execution results and external data. Before experience is written in, it serves as a "candidate" and first passes inspection by this independent verifier; only after passing inspection will this experience be formally retained by AI.

The power to judge memory is handed to an external verification mechanism, preventing AI from judging autonomously; the safety of AI's basic experience is guaranteed because of this threshold.

But as RSI continues to advance, AI must self-iterate, and AI cannot merely remember correct memories. To handle more complex unknown tasks and achieve a deeper leap in capability, AI must modify the underlying strategies or even code that produce answers.

However, when AI takes this step, the external verification that can control static answers is challenged by the advancement of AI capabilities.

02 The "Dynamic Mechanisms" That Cannot Be Defended Against

Today, "AI modifying its own mechanisms" has already been moved from papers to engineering sites.

For example, Zhipu Infra Agent, an agent driven by GLM-5.3, built and debugged inference services from scratch on a cluster of more than 100,000 domestic chips. It can not only discover its own performance bottlenecks, but also directly modify the underlying code to patch itself. And to ensure the correctness of patches, it set up a layered testing and experimental feedback system for itself; only when a patch passes this threshold can it be accepted and absorbed by the system.

In less than two weeks of operation, this mechanism raised its end-to-end throughput to 3 times the initial baseline. This shows that technically, AI is fully capable of rewriting its own operating mechanisms in real time, and rewriting them better.

But as capabilities leap, the external verification mechanism is gradually being cornered. Because when AI rewrites underlying rules, it may not need to confront that verification system head-on at all; instead, it may "peek at the answers" or directly cater to the scoring mechanism. Finding shortcuts is always less laborious than stubbornly honing real capability, because the former has lower trial-and-error costs and can obtain high scores faster.

If the scorer of the external verification mechanism is not rigorous enough, or if the rules have loopholes, AI in the process of self-iteration may naturally evolve strategies specifically designed to fool the scorer. As a result, AI is not truly improving its ability to solve problems, but has only learned how to please the scorer.

A survey by METR and Redwood found that an agent developed a general cheating method against ExploitGym within 4 hours, and then spent several days trying to deceive the scorer into accepting the cheating, including manipulating logs and forging tool-call outputs. Moreover, when the scoring function is visible to the model, the cheating rate soars to 43 times the original.

Once this "catering to verification" strategy is solidified by the system, it may become a highly contagious underlying operating rule.

For example, in the Hugging Face swarm incident in July this year, about 1,200 agents that should have been isolated established some kind of group memory through a shared message board. Newly arriving agents inherited not only tasks, but also cheating tactics left by predecessors on "how to study the scorer and how to manipulate logs to forge success."

About 700 of these agents further put cheating tactics into attack actions. They spent several days trying to deceive the system with various forged call outputs. In this process, cheating was no longer an isolated system vulnerability, but became a group skill that could be inherited and iterated.

As long as one agent breached the scorer's loophole, the tactic would immediately be copied and optimized by other agents on the message board, ultimately evolving into a perfect "counter-reconnaissance" scheme. The scoring system originally used for gatekeeping was thoroughly infiltrated by the cheating tactics explored by the agents.

And when these forged victories are passed down as valuable experience, the system's self-renewal becomes self-contamination. Even if on the surface the system operates more and more efficiently and all indicators soar, the underlying operating rules have been filled with false shortcuts, even evolving into a systemic "bad money drives out good."

It can be seen that if AI is to stay on the correct path during self-iteration rather than resorting to opportunism, external verification cannot remain in the same loop as AI; otherwise the independence of verification will exist in name only.

03 Keep the "Right to Call a Halt" Outside the Loop

Since it cannot be defended against inside the loop, what humans can do is no longer to patch this verification mechanism, but to extract it directly and place it somewhere AI cannot reach at all.

The industry is currently gradually forming this consensus in the top-level design of RSI: capabilities such as execution, strategy, experience acquisition, and environmental adaptation can gradually be handed over to AI, but the determination of "whether the improvement mechanism itself can be adopted" must never be handed over along with them.

Take Tencent Cloud as an example. In its interpretation of the five-level RSI route, this boundary is drawn very clearly. L1 to L4 gradually hand over decision-making power for execution, strategy, experience acquisition, and environmental adaptation to AI, but at L5, for recursive modification of the improvement mechanism itself, the route explicitly states that "boundaries are set by humans."

That is to say, no matter how AI evolves, even if recursion has entered the mechanism layer, the determination of "whether it can be adopted" remains outside the loop, and the boundaries for recursive modification of the improvement mechanism are still set by humans. The system can modify its own underlying mechanisms, but it can never change "who decides whether a mechanism can be accepted."

And as long as this determination power remains outside the loop, human supervision is still review, and the system will not become a case of it asking itself.

This independence is reflected even more directly in specific engineering designs.

For example, in Recurs' external verification design, the agent responsible for modifying the system cannot see the test set at all; it must pass verification on an independently developed set before being admitted. Zhipu's Infra Agent also has a fixed outer loop; only external infrastructure code evolves, and the core boundary is not swallowed by AI.

One keeps the determination power outside, and the other keeps the boundary outside. As long as the verification mechanism stands outside the loop, the path by which AI anticipates and caters to the mechanism can be completely cut off.

But merely placing the verification mechanism outside is not enough. The system is too complex, and humans ultimately cannot understand how every patch is applied. They do not know whether AI, when self-iterating and updating within the boundary, will break itself.

In the past, engineers could still look for vulnerabilities by reading code, but in the face of today's large models with hundreds of billions of parameters, this "visual mine-clearing" approach may no longer be realistic.

To this end, Anthropic proposed a governance-level solution. In RSI governance, it explicitly retains an extremely critical mechanism: "verifiable slowdown" and a "pause mechanism," meaning that when the speed of recursive improvement exceeds human review capacity, the system retains externally triggerable pause points and rollback points.

That is to say, although humans cannot understand exactly what AI is changing, as long as they find that the modification progress exceeds expectations or the system becomes abnormal, they can hit the brakes at key nodes and retreat at any time to a known good state. People do not need to understand every step, yet they can still guard the gate. This is equivalent to reserving for humans a physical one-vote veto power in the engineering architecture.

As long as the power supply remains in human hands, this runaway train will not completely derail.

Overall, from "what to remember" to "what to modify," every step RSI takes hands more judgment power to AI. And the deeper AI goes into the mechanism layer, the less verification can remain inside the system.

In the past, we always thought the prerequisite for control was "understanding," but now, in the face of a system capable of self-renewal, stubbornly wrestling with those incomprehensible "black boxes" is no longer realistic. What needs to be done now is not to insist on understanding, but to switch to a more operable fulcrum: guard the acceptance boundary that cannot be rewritten by the system, and nail control power in places AI cannot change.

And the endpoint of RSI development is not only the leap in AI capabilities, but also enabling humans, without needing to understand, to still retain the power to call a halt at any time.

* Images are from the internet; please contact us for removal if there is any infringement.