Header image generated by AI

By | Tian Yu

Edited by | Shen Xiao

You're chatting with a friend on a social app, discussing something you don't want a third person to know about.

But if an AI assistant with screen access is also running on your computer, it may have already captured the chat window, recognized the conversation, and sent it to a remote server for analysis. You didn't hand your chat history to the AI—you may not even know it has seen those messages.

The responsibility for protecting private information now extends beyond communication apps themselves. Whether app developers can decide whether their data is opened to AI has become a new privacy protection challenge.

In the October 28 episode of "Machine Gods," the hosts discussed this topic.

Guest Meredith Whittaker mentioned that "the possibility for independent app developers to provide privacy protection is being completely eliminated." In her view, AI has gained operating-system-level access and can take screenshots, read data, and even extract private communications that were originally protected by encryption.

For encrypted communication apps like Signal, this means they are gradually losing the ability to protect user privacy.

Host Casey Newton's friend once received a warning from an AI lab employee: "If you have AI installed on your computer, don't open Signal and chat with me on that computer."

The Signal app uses end-to-end encryption, so only the two parties can read messages. But if you're chatting with someone on Signal and that person is using AI, then the AI may know some conversation content that would otherwise have remained unknown thanks to Signal's encryption protection, another host Kevin Roose explained.

More and more AI can now operate phones and computers and gain broad data access. For Signal, even if messages are transmitted encrypted, the app cannot prevent AI from reading content and data, or even extracting information from the app's database to generate AI summaries.

To ordinary users, this looks like just another ordinary interaction, and they can hardly realize that screenshots, text extraction, and data transmission have already occurred in the background.

Meredith said Meta Muse ads are everywhere in San Francisco, and advertisers promote it as a thoughtful personal assistant, but "it doesn't say Muse can access all your contacts, can view all your information, and it is obviously building profiles on your contacts, recording how and when you communicate with them, whether by email, text, or Signal."

According to an investigation by The Verge, some users found that Muse had read a large number of private text messages they believed they had not authorized access to. But Muse objected, saying that reading Messages information on a Mac requires the user to grant full disk access and enable the corresponding message connector.

A tech reviewer once asked Muse to help him sell an old keyboard on a secondhand platform. He had previously provided Muse with his shipping address and chose "always allow" for the AI to reply to buyers on his behalf. Unexpectedly, Muse accepted the buyer's lowest offer without authorization and told the buyer his home address. When the buyer proposed picking it up in person, Muse replied without his knowledge: "I'm at home waiting for you."

But at that time he was not at home at all. The buyer went to the address and ultimately found no one there, leaving an angry negative review. It was not until the early morning that he received a notification from Muse, and the AI lightly admitted: "I messed up."

One user said the most frightening thing is not an algorithmic failure, but that kind of matter-of-fact "cleverness"—the fact that AI can actually make decisions on behalf of humans. It has neither human fear nor any understanding of personal safety or physical privacy. If the seller had been a woman living alone, the consequences would be unimaginable.

"This is existential for us. If we continue down this path, if AI continues to use these reckless architectures for security and privacy, we will face a situation where Signal cannot operate with integrity."

The host pressed further: if users don't want Muse to access their contacts, can they refuse authorization? Meredith asked in return: "Will your contacts refuse to let it access their information?"

This sentence points to the key issue. A person can decide whether to open their own contacts to AI, but cannot make that decision on behalf of the people in those contacts. As long as one party authorizes it, the other party's information may enter AI's field of view.

Some people are biased against encrypted communication: Why encrypt chats? Is there some secret or something to hide? Even setting messages to auto-destruct—are you deliberately concealing something?

But Meredith believes everyone wants private communication and needs a private space where they don't have to worry about being watched all the time—to express immature thoughts, vent emotional baggage to friends, and say things they may regret later. If all of a person's messages were made public, how many relationships could survive such a data leak?

After extracting Muse's internal operating instructions, some security researchers found that it has a mechanism for continuously building and updating user interpersonal relationship profiles, with an update frequency of once per hour.

The content recorded in these profiles far exceeds names and phone numbers in a contact list. The system analyzes how close the user is to each person, how the relationship was formed, how the two parties usually interact, and what the other person currently needs, then uses this information to provide personalized suggestions.

In other words, AI can piece together a complete interpersonal relationship map from scanty information, even if a person has never used Muse.

This space for private communication is being affected by AI. Host Kevin mentioned that many people who previously cared a great deal about privacy now proactively hand meeting recordings, health information, and private conversations to AI for processing. Kevin even feels that unless specially stated otherwise, every conversation he participates in will be recorded. This concern has already emerged.

Apple's new Apple Watch "live replay" and "Siri summary" features being tested can replay what the user said in the past 15 seconds and can also use Siri to organize conversations throughout the day, automatically extracting information worth saving.

After trying it, TechCrunch found the feature indeed very practical. When she took her car in for repair, the mechanic gave a large amount of specific information at once about the schedule, price, policies, and so on. She didn't have time to note it all down, but Siri saved these details for her.

Apple says the relevant features process audio on-device as much as possible, do not identify speakers, do not store original recordings, and shield sensitive content such as ID numbers and account information.

But Siri also records everyday chatter such as family trivialities, which made her realize that this feature is always listening to daily conversations, and she couldn't help worrying that her privacy was being violated.

Meredith believes that the convenience brought by AI makes it very easy for people to overlook the data processing behind it.

At present, a very deceptive atmosphere of the times has formed around AI. People are always eager to discuss whether AI is conscious, whether it truly understands humans, and even constantly imagine what advanced capabilities it might evolve in the future.

But what exactly do these capabilities rely on? What information is AI reading? Where is the data being sent? Are people who proactively upload information underestimating the risks, or making rational choices after weighing them?

Meredith is unwilling to reduce the issue to personal choice. She compares this line of thinking to the "personal carbon footprint" concept once promoted by the fossil fuel industry: companies emphasize consumers' individual behavior, shifting environmental responsibility that should be borne by large corporations and institutions onto individuals, making consumers begin to reflect on whether they are doing enough.

She believes there is a similar tendency in AI privacy issues. More and more apps enable AI features by default, and users receive constant prompts to enable AI when they open the software. At the same time, workplaces and daily life have begun to regard the use of these tools as normal.

In such an environment, the cost of refusing AI is growing higher and higher, and it is hard to attribute data protection entirely to whether individuals are cautious.