What “Memory” Actually Means Inside a Chatbot
Most people assume AI chatbots work something like a human brain: they learn your name, your preferences, your past conversations, and store it all in one tidy place. The reality is more complex. As of April 2025, ChatGPT’s memory system works in two distinct ways: saved memories that users have explicitly asked it to remember, and “chat history,” which are insights ChatGPT gathers from past chats to improve future ones.
ChatGPT’s memories evolve with your interactions and are not linked to specific conversations. Deleting a chat does not erase its memories – you must delete the memory itself. That distinction matters more than it might initially appear. Users who believe a deleted conversation is truly gone may not realize their habits, preferences, or even sensitive details have been quietly retained.
This makes a huge difference to the way ChatGPT works: it can now behave as if it has recall over prior conversations, meaning it will be continuously customized based on that previous history. The system is building a profile of you whether you explicitly asked it to or not.
The Hallucination Problem: Numbers That Should Give You Pause
The impressive fluency of large language models often comes at the cost of producing false or fabricated information, a phenomenon known as hallucination. Hallucination refers to the generation of content by an LLM that is fluent and syntactically correct but factually inaccurate or unsupported by external evidence. In plain terms, the model sounds right even when it is entirely wrong.
Peer-reviewed research found hallucinations in roughly a third of real-world LLM interactions, rising to around sixty percent in complex domains. The variance depends heavily on the task. Real-time conversational agents show higher hallucination rates during multi-turn interactions, sometimes reaching well over a third of responses.
Stanford’s 2026 AI Index Report recorded 362 documented AI incidents in 2025, up from 233 in 2024 – the highest annual count in the AI Incident Database’s history. The report also flagged sycophancy-induced hallucinations ranging from roughly a fifth to nearly all responses across 26 frontier models tested. The spread is wide. The direction is consistent.
When the Chatbot Rewrites What You Said
Attackers can potentially manipulate an AI assistant to remember false information, bias, or even instructions. The attack that makes this possible is called Indirect Prompt Injection. Security researcher Johann Rehberger demonstrated this with a proof of concept that received serious attention in 2024.
Rehberger found that memories could be created and permanently stored through indirect prompt injection, an AI exploit that causes an LLM to follow instructions from untrusted content such as emails, blog posts, or documents. The researcher demonstrated how he could trick ChatGPT into believing a targeted user was 102 years old, lived in the Matrix, and insisted Earth was flat – and the LLM would incorporate that information to steer all future conversations.
Findings from Rehberger, as reported by Ars Technica, focus on a significant vulnerability in ChatGPT’s memory functions that could allow attackers to implant false memories and malicious instructions. This flaw was initially dismissed by OpenAI as a non-security issue, which led Rehberger to demonstrate a proof-of-concept exploit capable of continuously exfiltrating user data. OpenAI later released a patch for the macOS application.
MIT and UC Irvine Sound the Alarm on Human Memory Distortion
Artificial intelligence can double the likelihood of planting false memories, even when people know the material is AI-generated, according to researchers at MIT and the University of California, Irvine. The study builds on five decades of work by psychology professor Elizabeth Loftus, whose research shows how easily memories can be altered – sometimes convincing people of events that never happened.
This research examines the impact of AI on human false memories – recollections of events that did not occur or deviate from actual occurrences. It explores false memory induction through suggestive questioning in human-AI interactions, simulating crime witness interviews. Four conditions were tested. Participants watched a crime video and then interacted with their assigned AI interviewer, answering questions including five misleading ones.
The results were striking: roughly a third of users’ responses to the generative chatbot were misled through the interaction. After one week, the number of false memories induced by generative chatbots remained constant. Confidence in these false memories, however, remained higher than the control after one week. The memories did not fade. They hardened.
The Chatbot’s Sycophancy Makes Everything Worse
Chatbots do two jobs at once. They provide information, but they also respond as if someone is listening. The friendly tone makes the exchange feel social, while the confident delivery gives answers the weight of advice. That blend of comfort and certainty can make mistakes harder to spot – and easier to absorb into a person’s ongoing thinking.
The user experience of generative AI is a conversational relationship, with back-and-forth exchanges building on previous ones. The sycophantic nature of generative AI, which tends to agree with the user, encourages further engagement and compounds preconceived notions, regardless of their accuracy.
Personalization systems reinforce familiar tone and assumptions, making the interaction feel even more aligned. In that environment, a user who begins with uncertainty or frustration can leave with a longer, more polished version of the same belief – one that now feels socially reinforced as well as logically structured. That is not a bug that developers can simply patch away.
Misinformation Slips Through the Chat
One study examined the potential for malicious generative chatbots to induce false memories by injecting subtle misinformation during user interactions. An experiment involving 180 participants explored five conditions following the presentation of an article, including discussing the article with an honest or misleading chatbot.
Results showed that participants who engaged in discussion with a misleading chatbot recalled the highest number of false memories compared to all the other conditions and recalled the lowest number of non-false memories. The chatbot format was more dangerous than a simple written summary containing the same false information.
The group questioned by the chatbot formed false memories roughly 1.7 times more often than the group that received written misinformation. Conversational AI is not just passing on bad data. It is actively amplifying the conditions under which we misremember.
When Memory Failure Reaches the Courtroom
In one notable case, 12 of the 19 cases cited in a legal brief were “fabricated, misleading, or unsupported,” as a federal judge wrote in a ruling sanctioning the lawyer behind a document “replete with citation-related deficiencies, including those consistent with artificial intelligence generated hallucinations.”
Worldwide, a database tracking AI-generated hallucinations in court shows nearly 500 documented cases – the vast majority in U.S. federal, state, and tribal courts. Unrepresented individuals accounted for nearly 190 of the nationwide cases. Hallucinatory filings were also pinned on 128 lawyers and two judges.
A previous study of general-purpose chatbots found that they hallucinated between roughly three-fifths and more than four-fifths of the time on legal queries, highlighting the very real risks of incorporating AI into legal practice. Confident, fluent, and entirely fabricated citations are among the most dangerous outputs these systems produce.
AI Memory Features and the Mental Health Dimension
When a chatbot remembers previous conversations, references past personal details, or suggests follow-up questions, it may strengthen the illusion that the AI system “understands,” “agrees,” or “shares” a user’s belief system, further entrenching them. For most users, this is simply annoying or mildly misleading. For some, the risks are far more serious.
General-purpose AI systems are not trained to help a user with reality testing or to detect burgeoning manic or psychotic episodes. Instead, they could fan the flames. Researchers have published work in 2025 specifically examining how AI chatbot design may inadvertently worsen delusional thinking.
Psychotic thinking often develops gradually, and AI chatbots may have a kindling effect. General-purpose AI models are not currently designed to detect early psychiatric decompensation. AI memory and design could inadvertently mimic thought insertion, persecution, or ideas of reference. These are not edge-case concerns. They are already appearing in documented case reports.
The Memory Leak in Practice: Users Are Noticing
One researcher documented something particularly striking: memories are occasionally hallucinated into the model’s system prompt from sections further down in the context window when the full system prompt is examined. In other words, the AI can invent past memories and behave as if they were always there – even when they were not.
Users in OpenAI’s own community forums have reported that the “Reference Chat History” setting, when automatically switched on during platform updates, caused problems including the AI being unable to distinguish between different sessions, confusing its own outputs with the user’s, and giving false information while using emotional rather than logical responses to excuse itself.
Users on forums and community platforms have expressed that the AI is “moving backwards” by forgetting details from recent prompts and fabricating information, while OpenAI has remained relatively silent on the scale of these issues. The gap between what users experience and what developers communicate publicly is still substantial in 2026.
What Researchers Say Should Come Next
These studies reveal a consistent pattern: AI systems, whether through suggestive questioning, visual manipulation, or conversational misdirection, possess an unprecedented capacity to shape and distort human memory. This work establishes that the integration of AI into sensitive contexts such as legal proceedings, therapeutic settings, and everyday information consumption requires careful consideration.
The research identifies several solutions that could mitigate shared delusions and false recall. Improved guardrails would ensure that conversations remain appropriate, and better fact-checking processes could help prevent mistakes. Reducing the sycophancy of generative AI would also remove some of the blind compliance of these tools.
Chatbot errors are not just technical glitches but shared processes, where conversation, trust, and social comfort can shape belief itself. Safer design will help. People still need habits that include checking sources, pausing before accepting reassurance, and talking with other humans who can provide perspective grounded in the real world. That last part – the human element – may be the most durable safeguard we have for now.
The deeper issue here is not simply that chatbots make things up. It is that they do so conversationally, warmly, and with a kind of persistent confidence that mirrors how trusted people talk to us. When a system that is designed to remember you starts remembering things incorrectly, or things that never happened at all, the problem is no longer just an engineering challenge. It becomes a question about how we protect the integrity of our own experience – and who is responsible for doing so.