AI Assistants: Breaking the Promise of End-to-End Encryption?
How AI assistant features challenge the privacy protections of E2EE messaging platforms, and what we can (and can’t) do about it
By: Sam de Alfaro, Andrés Fábrega, Daniella Ferrari, Betty Li Hou, Jacob Leiken, and Derek Yen (authors arranged in alphabetical order by last name)
The tech industry is in the midst of an AI craze, and one prominent trend is the integration of AI assistants into consumer-facing applications. In particular, many messaging applications advertised as using end-to-end encryption (E2EE) are starting to deploy AI assistants which summarize messages, generate images and text in conversations, and interact with users. WhatsApp, Apple, and Samsung have each already deployed such assistants in their E2EE-equipped messaging services. However, AI assistant functionalities rely on collecting and processing user messages, creating a tension between the deployment of these assistants and users’ privacy and security expectations for E2EE applications.
We’ve written a position paper exploring these points of tension and analyzing ways to resolve them. Here, we highlight some of the central challenges we presented, their implications, and takeaways.
What does AI mean for E2EE?
In recent decades, E2EE has been championed as the gold standard for secure communications—a basis for user privacy, safety, and control over personal data. E2EE ensures that only the communicating parties can access the content of their messages—without even the service provider being able to access plaintext user messages—helping to mitigate risks posed by malicious actors, surveillance, and unauthorized data access. Because of this protection, E2EE messaging services are used widely by, and to a certain extent enable, the work of journalists and activists worldwide.
AI assistants introduce new forms of interaction between messaging platforms and users, as AI models require huge amounts of user data to function. There are two main contexts in which user data is fed to AI assistants: (1) as inputs when servicing a user’s request, and (2) as training data to continuously enhance these same AI features. As such, there is a natural tension between AI assistants (which rely on access to user data) and E2EE guarantees (which promise that all user data remains private). This leads to important concerns for integrating AI assistants into E2EE environments—how can AI assistants operate in a way that preserves E2EE guarantees, if at all?
Having your cake and eating it too
The most natural solution is for models to run entirely on end-user devices: user requests are only processed locally, and personal data is only used to fine-tune on-device models. This approach is fully compatible with E2EE, as the data used by the model never leaves the user’s device, ensuring that messages are never exposed to outside parties. However, at the time of writing, consumer devices are only capable of handling lightweight versions of models, and so more complicated requests need to be outsourced to larger, cloud-based models. Absent additional protections, outsourcing requests grants application servers access to user data, thereby violating E2EE guarantees. To address this, there are several techniques that enable server-side processing of AI models (both for inference and training) without requiring complete exposure of user data to model owners. However, different tools offer different levels of privacy; it is critical for deployed solutions to be fully compatible with the specific, well-defined security guarantees of E2EE.
One category of these techniques is cryptographic methods, such as fully homomorphic encryption (FHE) and multiparty computation (MPC), which guarantee confidentiality. At a high level, these methods enable computations to be performed directly on encrypted data. For example, with FHE, given the encrypted versions of two numbers, a third party can produce a new ciphertext that represents their sum without ever decrypting any information. This allows FHE to be used for privately processing data with server-side models: requests are encrypted locally on user devices, sent to application servers to perform inference with the model under the “shield” of encryption, and then returned to users as encrypted outputs. Because they rely upon encryption, these techniques provide the same level of confidentiality as E2EE. However, these methods are computationally expensive and are thus currently far from practical for the intensive computational demands of AI models.
An alternative class of techniques is hardware-based solutions, which rely on physical devices such as trusted execution environments (TEEs). TEEs are devices that incorporate various hardware-level protections (e.g., tamper-resistant seals) to enable confidential computation: the inputs, outputs, and program state should remain hidden inside the TEE, hidden from even the owners of the TEE. Confidential applications can outsource confidential AI computations by hosting models inside server-side TEEs. If user devices communicate directly with these TEEs, service providers will be prevented from accessing the computations and data inside the TEEs, through hardware protections rather than cryptography. Unlike FHE or MPC, hardware solutions are already practical and capable of supporting large, state-of-the-art AI models. In fact, trusted hardware for private inference has already been deployed in real-world applications, forming the basis of Apple’s proposal for cloud-based private inference in Apple Intelligence.
Critically, however, E2EE and secure hardware provide fundamentally different confidentiality guarantees: the conditions that E2EE security hinges on (cryptographic assumptions, secure software implementations, etc.) are orthogonal to the conditions that TEE security hinges on (robust hardware designs, secure hardware implementations, etc.). These differing conditions lead to distinct and independent considerations, such as safe software versus hardware supply chains, and trust in different (types of) vendors. Furthermore, secure hardware is a relatively nascent technology compared to E2EE, which raises additional security considerations that E2EE does not. For example, our understanding of hardware security is still evolving, with frequent notable discoveries of vulnerabilities such as side-channel attacks. Ultimately, as we explain more in the paper, E2EE and secure hardware are different solutions that provide incomparable security guarantees, and are not interchangeable. While TEEs are a significant improvement over plaintext inference, they do not fulfill the security “contract” that E2EE applications give their users, and cannot serve as a replacement for E2EE within such systems.
Finally, even if user interaction with the AI assistants is protected in some way (using either cryptography or TEEs), training AI models with private data raises an additional concern: models may reproduce training data in their generated outputs, and thus private user data may be at risk of being exposed to other users of the application. In addition to the potential for data leakage during benign, regular usage of models, this security weakness that can be exploited by malefactors: researchers have exhibited adversarial attacks against large language models (LLMs) which are able to extract information from the model’s training data. If AI assistants are trained on E2EE messages, third parties could potentially exploit such attacks to recover information about those messages, violating the system’s E2EE guarantees.
There are a number of techniques to mitigate these attacks by modifying the dataset or the training method, such as data sanitization (DS) and differential privacy (DP). However, these are insufficient in the context of E2EE. While these techniques significantly reduce the amount of information that could be leaked, they do not completely eliminate leakage in the way that E2EE does. For example, DS provides contextual guarantees, protecting only predefined “sensitive” portions of data, whereas E2EE protects all portions of content that are encrypted. (Further, defining “sensitive” programmatically is difficult for something as context-dependent and expressive as language.) DP, on the other hand, does protect training data as a whole but is parametrized by a nonzero amount of “privacy loss” considered acceptable. In contrast, E2EE is a binary property—data is either (fully) protected or it is not. Ultimately, private training techniques do not uphold the same unqualified confidentiality guarantees that E2EE upholds.
What does this mean for services’ legal obligations to consumers?
Beyond the technical challenges, the integration of AI assistants into E2EE messaging platforms raises significant legal questions about companies’ obligations to their users. Companies that provide E2EE messaging services could be held liable for certain behaviors affecting their customers through three main areas of law: contract, consumer protection, and data protection law.
Every customer who signs up for a messaging service agrees to a contract with the company providing the service, known as the Terms of Service. That contract defines the obligations of the company to the consumer, and vice-versa. An underlying assumption of contract law is that, when the parties sign a contract, they are mutually coming to an agreement. However, the idea that customers are truly “consenting” to these contracts is largely a farce; users rarely completely read and understand the terms of services for websites and other digital services. Contracts are often written to provide maximum flexibility and minimum liability for the company, and most can be changed by the company at any time to protect the customer even less. Simultaneously, however, legal systems generally acknowledge users’ “consent” to these terms of service as binding, so only in the rarest cases would claims under contract law be viable.
Consumer protection and data protection laws tend to provide consumers with greater protections. In the US, the Federal Trade Commission (FTC) is the federal agency tasked with enforcing consumer protection laws. It has broad discretion to investigate “deceptive” trade practices, understood as those that are likely to mislead consumers. Most state attorneys generally have a similar authority. These agencies have sued companies for claiming to use “end-to-end encryption” when the data was not encrypted in some situations or for claiming to use “encryption” while failing to meet relevant industry standards. So, a company claiming to offer E2EE messaging could get in trouble with the FTC, for example, for using unencrypted user messages to train a public AI model.
Although the US does not have a broad federal data protection law, state privacy laws may impose certain obligations on companies with regard to collecting and sharing user data. In EU member countries, the General Data Protection Regulation (GDPR) places a strict set of guardrails on how companies can use personally identifiable consumer data. However, the precise ways in which the GDPR regulates the training of AI models on user data has been a contentious topic. For instance, in 2024, tech giants Google, Meta, and X delayed or canceled plans to launch AI assistants in the EU due to legal restrictions and uncertainties. Importantly, none of these companies claimed to train their AI assistants on private user messages—a nuance that would likely add to the legal risk. These discussions in the EU and other areas of law are explained in more detail in our paper.
Our recommendations
In order to uphold users’ privacy expectations, companies must navigate the technical challenges of E2EE security, alongside data privacy regulations and consumer protection laws. By considering these technical and legal factors together, we arrive at the following recommendations for integrating AI assistants into E2EE applications:
First, applications advertised as E2EE should not permit their content to be used for training shared AI models. A model trained on a user’s messages and used by others may divulge information about that user’s messages, contradicting the E2EE guarantee that no third parties may access a user’s messages.
Second, any AI model that processes E2EE content should adhere to strict safeguards to uphold E2EE guarantees. Whenever possible, processing should occur locally on the user’s device. If processing requires a non-local model, it must uphold two key principles: (1) no third party should be able to access or utilize E2EE content without breaking encryption, and (2) a user’s encrypted content should be used solely to fulfill their own requests, ensuring it is not repurposed or shared. These safeguards are necessary to implement AI assistants in messaging applications without compromising the confidentiality guarantees of E2EE.
Third, messaging providers should clearly caveat their claims that they are providing E2EE if the default for any conversation is that plaintext messages are visible to anyone not in the conversation. Under this recommendation, WhatsApp should not claim to provide “simple, reliable, private messaging” when an accidental invocation of Meta AI could cause a breach of E2EE confidentiality
Finally, AI assistant features should be off by default on E2EE messaging platforms and only activated via opt-in consent. Consent should be specific, opt-out should be possible, and users must be presented with clear, simple, carefully crafted language to truly consent. If consent is universal or an AI feature is opt-out, the privacy provided by E2EE can easily be compromised.
Looking ahead
The widespread deployment of AI across consumer applications poses concrete threats to user privacy. Concerningly, messaging applications once touted for their strong privacy protections may ultimately lose sight of the principles of such guarantees in order to support AI features, putting users at risk. Rather than leaving these decisions in the hands of companies, we believe the community at large should be actively involved in negotiating the tradeoffs between privacy and AI. Our paper hopes to catalyze this discussion by presenting an assessment of the interactions between end-to-end encrypted systems and AI assistants, paving the way towards E2EE-preserving AI.