Even for large companies, internal control systems need to be strengthened during the development of AI

Can large models also be 'poisoned'?

Recently, it was reported that ByteDance's large model training was 'poisoned' by an intern.

According to reports, the incident occurred in ByteDance's commercialization team. Due to dissatisfaction with the team's resource allocation, intern Tian used a vulnerability in HF (Hugging Face) to inject malicious code into a shared model, resulting in damage to the team's model training results. It is reported that the compromised code had been injected into more than 8,000 cards, causing losses of up to tens of millions of dollars.

However, in the afternoon of the 19th, ByteDance officially clarified that while it is true that a serious disciplinary violation occurred with an intern in the commercial technology team, reports claiming that ByteDance's large model training was attacked by an intern were exaggerated. The intern has since been dismissed.

While the incident did occur, some of the related reports contained exaggerated and inaccurate information. The intern maliciously interfered with the model training tasks of the commercial technology team's research projects, but this did not affect the formal commercial projects or online business, nor did it involve ByteDance's large model or other businesses. Additionally, the reported "involvement of over 8,000 cards and losses of tens of millions of dollars" was significantly exaggerated.

Upon verification, it was confirmed that the intern in question had been interning with the commercial technology team and had no experience interning at AI Lab. There were inaccuracies in their social media profile and some media reports. It is understood that the intern was dismissed by the company in August and the company has reported their behavior to industry associations and the school where they are studying for further action.

However, this incident also exposed issues with security management in ByteDance's technology training, including permission isolation and shared code auditing. According to Jiu Pai News, a technical security expert stated that the HF vulnerability likely stemmed from a lack of permission isolation and inadequate auditing of shared code. Under normal circumstances, significant code changes must undergo auditing, with records kept of any operations performed, making it impossible for a single individual to modify the code without detection.

Editor: Ding Li