In order to comply with data protection regimes, AI chatbots and associated machine learning applications will have to be capable of forgetting what they’ve learned.
It’s not yet evident they can handle that requirement.
Researchers affiliated with Australia’s National Science Agency (CSIRO’s Data61), and Australian National University – Dawen Zhang, Pamela Finckenberg-Broman, Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu – recently issued a paper on the subject.
Citing the “right to be forgotten” or right to erasure under Europe’s General Data Protection Regulation (GDPR), the academics argue that large language models, such as OpenAI’s ChatGPT, Google’s Flan-T5, Meta’s LLaMA, and Anthropic’s Claude, and the applications integrating these models (Microsoft Bing, GitHub Copilot, Google Bard, and third-party apps linked via API) will find compliance challenging because they process and store information in a way that’s different from search engines.
It’s not just the EU’s GDPR that promises this limited right. The California Consumer Privacy Act (CCPA), Japan’s Act on the Protection of Personal Information (APPI) [PDF], and Canada’s proposed Consumer Privacy Protection Act (CPPA) have data deletion or correction provisions of sorts. And there’s also the EU’s new AI Act to consider.
Real-world effects
The potential for legal entanglement is not merely theoretical. In March, Italian authorities temporarily suspended access to ChatGPT on the grounds that it failed to comply with data protection rules, before relenting the following month. That same month, the Office of the Privacy Commissioner of Canada opened an investigation into ChatGPT data compliance. The investigation was expanded the following month and remains ongoing. France, and Spain are also conducting inquiries.
The Australia-affiliated academics observe that while the right to be forgotten was initially applied to Google Search, it’s relevant to large language models, and not just because they’re being used to augment services like Microsoft Bing and Google Search.
Search engines, the authors say, have evolved over the years but continue to be structured around a document index-linked to search keywords. Identifying specific data and making it inaccessible or removing it is relatively straightforward.
“In contrast, in LLMs, it is hard to know what personal data are used in training and how to attribute these data to particular individuals,” the researchers say. “Data subjects can only learn about their personal data in these LLMs by either inspecting the original training dataset or perhaps by prompting the model.”
One problem, they say, is that training datasets may not be disclosed. Another is that prompting trained models to see how they respond doesn’t guarantee the text output contains the entire list of information stored in the model weights that affect the output. Then there’s the issue of hallucinated data – supposed facts just made up by an AI bot – which the researchers say cannot be accessed reliably.
Whereas a search engine can either remove an offending web page from its index, or delist links associated with personal data, those methods don’t apply to LLMs, the boffins say.
Removing personal data from an LLM’s training dataset doesn’t affect existing trained models, they note, and building a new version of the model can take several months – more than the delay allowed under GDPR. And that’s to say nothing of the expense of training LLMs.
Removing data from a trained model is difficult, though ways to do so are being explored. For example, there’s a “machine unlearning” [PDF] technique called SISA, short for Sharded, Isolated, Sliced, and Aggregated training. There’s also Inductive Graph Unlearning and Approximate Data Deletion, among other approaches to oblivion.
The boffins from down under, however, point out that not everything one might want to remove from a model can be found within it.
“Hallucinated data is not contained in the training dataset of the model, and hallucinated data from the model is hard to eliminate,” they say in their paper. “Even if some hallucinated data could be removed from the model, side effects and new hallucination might be introduced. Eliminating hallucination from LLMs is still impossible now.”
Hallucinated data from the model is hard to eliminate
It may be that the right to be forgotten is fundamentally at odds with the technical persistence of data memory, at least until bit rot sets in. In 2018, Boston University researchers published a paper [PDF] titled “Humans Forget, Machines Remember: Artificial Intelligence and the Right to Be Forgotten.”
They concluded, “The Right to be Forgotten may very well be a well-intentioned regulatory protection, and many would argue that it is an important right to be protected. However, there is a clear disconnect here between law and technical reality.”
Nonetheless, those making AI models are attempting to bridge that gap. In late April, OpenAI, maker of ChatGPT, published limited information outlining how it trains models and how that training complies with privacy laws. The company concedes that ChatGPT may include personal information and provides an email address, dsar@openai.com, for handling data subject access requests.
“Individuals in certain jurisdictions can object to the processing of their personal information by our models by filling out this form,” the company said. “Individuals also may have the right to access, correct, restrict, delete, or transfer their personal information that may be included in our training information.”
It’s not immediately clear how OpenAI handles data removal requests or how long such requests take to implement. The company may, for example, simply create data masks or guardrails that block certain output patterns. Or it may collect removal requests and batch process them periodically when its models get re-trained. Or it may take another approach.
OpenAI did not immediately respond to a request for comment. Google, Meta, and Microsoft also did not immediately respond. ®
