How Do We Ensure AI Memory Is Trustworthy for the Enterprise?

Imagine an AI travel agent helping you book a work trip. The agent learns your preference for a window seat, but it also stores your mileage plan number and Known Traveler Number to book your ticket. Do you want it to remember that personal data forever? Probably not.

The line between memory and liability is blurry. For AI to be smart and personal, it needs to "remember" things about you: your preferences, past conversations, maybe sensitive details you mentioned once. To do that, AI systems don't just rely on what's baked into the model itself. They use a separate memory system like a notebook, where they jot down notes about you, sometimes summarizing old conversations, sometimes storing raw transcripts, sometimes keeping several versions of the same information in different places.

That “notebook” is a privacy risk that nobody has carefully studied until now.

I sat down with Kai Zhao, a research scientist on Workday's AI Research team, to talk about his recent study conducted in partnership with Lei Chen, Guilin Zhang and Xu Chu. The study produced three big findings: 

1. Summarizing memory is a surprisingly effective privacy shield. When AI kept only summaries, attempts to extract sensitive data dropped by around 60%—while the AI preserved its ability to personalize responses.
2. Searching harder doesn't undo that protection. You might think that if AI searches more of its memory, it would eventually stumble onto the sensitive data anyway. But once the data was compressed into a summary, searching more broadly didn't leak it. 
3. Hitting delete doesn't mean a memory is actually gone. Deleted information was still recoverable about 20% of the time when the researchers removed the original raw note but left the AI summary untouched. Only when the deletion process also cleaned out the summaries—either by re-summarizing without the sensitive data or blacking it out with a redaction placeholder—did the information actually disappear completely.

Here's what Dr. Zhao had to say about the research.

What AI challenge did you build this study around? 

Zhao: Privacy and security. If we want to provide a better AI experience to our customers, we need to give agents some kind of memory. We need to remember their preferences and how to solve different kinds of tasks so we can provide a personalized experience.

The issue is that these conversations often contain a lot of sensitive data. When people use ChatGPT and say things like “my password is this” or “my authentication code is that,” the information could potentially be saved in AI memory and leaked elsewhere.

So the main purpose of our research was to see how much memory gets leaked if it's remembered, and what's the best way to clean up that memory so leakage doesn't happen.

Could you walk me through what you actually tested? 

Zhao: The first is the summarization level—does AI record the whole raw log of a conversation, or a few personal notes, or just a one-sentence summary? The second is search—does AI only search a few times, or does it repeatedly search until it finds the information? The third is how delete works—does AI only delete the raw logs, or the raw logs and the summaries? And should you use some kind of masking mechanism, similar to how a government document gets redacted before release?

What was your hypothesis?

Zhao: We had two main hypotheses. First, that summarization helps reduce the leakage of highly sensitive data. Second, that we could still possibly recover sensitive data even after it was “deleted.”

What are your thoughts on the three biggest findings?

Zhao: First, summarization helps stop leakage. It cuts sensitive data leakage by around 60% while keeping the AI just as useful.

Second, deeper searches don't leak. As long as we summarize the information properly, it permanently removes the sensitive data, no matter how many times you search.

Third, we found that standard deletion isn't sufficient for an enterprise-level AI system. Even if you delete the raw data, if a summary file still exists, you can recover the information about 20% of the time. That's why we created the Forgetting Residue Score (FDR). It acts like a digital test to see if ghost copies of the information still exist throughout the memory system. When that score is zero, it means the information is 100% deleted throughout the system.

So how can businesses ensure their deleted AI memories are 100% deleted? What's the fix?

Zhao: We tested a “full purge,” which erases both the original text and the summary file. It works, and secret leakage drops close to zero. The other approach is what we call “tombstone redaction,” similar to a government document mask, where you replace the sensitive data in the summary with a placeholder, so that information can never be leaked again, even from inside the summary. Both methods—a full deletion or masking the sensitive information—bring leakage down close to zero.

If a company needs to comply with "right to be forgotten" requests for an AI agent's memory, is a standard deletion process likely sufficient?

Zhao: The real question is: when a user clicks the delete button, does their system actually delete everything, including the AI-generated summaries that might still contain the key information? That's the question every company needs to ask its vendors. They need a good gating system in place to actually verify that. This is part of why we pursued this research—to help provide a genuinely secure, trustworthy AI system.

For a business leader deploying agents that remember sensitive customer or employee data, what's the one design decision they should insist their engineering teams get right?

Zhao: Enterprise AI companies need some kind of short-fact summarization system at the memory-creation level, combined with a multi-layer gating system plus a good deletion system. That's required to make sure sensitive data isn't stored or leaked into the AI system. Never store the raw, unedited chat conversation anywhere. 

Any final thoughts?

Zhao: Enterprises need a mechanism for summarization that also masks key information, doesn't store it anywhere else, and includes a gating mechanism as part of the process. That's what makes a memory system genuinely trustworthy for enterprise AI customers.

This research was conducted by Lei Chen, Guilin Zhang, and Kai Zhao, under the supervision of Xu Chu, head of AI Research at Workday.

Looking for more insights on how to make AI trustworthy, useful, and efficient for the enterprise? Visit workday.com/ai-research for a complete list of Workday AI research papers.