Categorizing AI Memory

I've been trying to code up my own AI agent from "scratch" in order to learn how they work and what problems people have solved or still have to solve. Long-term memory turned out to be tricky. Like, I want the AI to remember things I've talked about in the past, things it researched independently, and stuff like that.

RAG (retrieval-augmented generation) is what is usually used for things like long-term memory, and while I have had success in getting RAG to find things like factual documents, it hasn't been very precise picking out episodic memories or personal facts.

Which is certainly a skill issue, and one I'm learning to resolve via experimentation, though it did put me down a rabbit hole to see how people have been trying to enhance the memory of AI systems.

Inspired by the paper A Survey on Memory Mechanisms in the Era of LLMs by Yaxiong Wu et al, I created a visualization of the most prominent examples of AI memory systems being developed in the wild.

Diagram Transcript

1. Quadrant I: Conversational Memory
   • ChatGPT (CG): Dialogue stored as "user" and "assistant". The conversation transcript IS the short-term memory.
   • DeepSeek (DS): Dialogue stored as "user" and "assistant". The conversation transcript IS the short-term memory.
   • Claude (Cl): Dialogue stored as "user" and "assistant". The conversation transcript IS the short-term memory.
2. Quadrant II: Episodic Memory
   • MemoryBank (MB): Builds a profile of a user based on conversation history, and uses the Ebbinghaus Forgetting Curve to reinforce memories by recency and importance.
   • A-MEM (AM): Uses the Zettelkasten notetaking method to create knowledge networks, associating memories with each other.
   • HippoRAG (HR): A RAG-based system that constructs knowledge graphs over entities and phrases, as well as concepts.
   • RMM (RM): Reflective Memory Management: combines Prospective Reflection for summarization with Retrospective Reflection for retrieval.
   • RET-LLM (RL): Monitors and updates its factual knowledge of the world in real-time.
   • LD-Agent (LD): Uses persona modeling to build profiles of users and agents.
3. Quadrant III: Contextual Memory
   • Prompt Cache (PC): Precomputes frequently requested conversation history to save time.
   • Contextual Retrieval (CR): Applies prompt caching to RAG to reduce the overhead of generating contextualized chunks.
4. Quadrant IV: Identity Memory
   • Character-LLM (Ch): Trains LLMs to role-play historical figures (such as Cleopatra) using records of their experiences.
   • AI-Native Memory (AN): Argues that each person should have a personal LLM model (called LPM, or large personal model) as opposed to long text for memory.
   • MemoRAG (MR): Compresses a user's entire body of knowledge into a parametric KV cache that the LLM uses to enhance a normal RAG query.
   • Echo (Ec): An LLM fine-tuned to reason temporally, using primarily episodic data.
5. Quadrant V: Working Memory
   • ReAct (RA): Reasons by verifying thoughts with actions (searches, calculations), grounding the thoughts in reality.
   • Reflexion (Rx): When it fails, it assesses the failure and reattempts using the assessment as additional context.
6. Quadrant VI: Procedural Memory
   • Voyager (Vy): Creates a reusable skill library over time based on external feedback.
   • Buffer of Thoughts (BT): Stores thought patterns into a memory repository that it later recalls in order to enhance chain-of-thought reasoning.
   • Agent Workflow Memory (AW): Chooses one of several reusable workflows to guide reasoning on a task based on the task's goals.
   • Think-in-Memory (TM): Generates new thoughts after thinking (post-thinking) that it later recalls when relevant.
   • Retroformer (Rt): Keeps reflections of failures as long-term memory that it uses as context for future queries.
7. Quadrant VII: Attention Memory
   • vLLM (vL): A high-efficiency serving system built on PagedAttention, a virtual-memory-inspired mechanism enabling flexible sharing across requests.
   • ChunkKV (CK): Compresses the KV cache by grouping tokens into semantic chunks, keeping the most informative ones and enabling layer-wise index reuse.
   • LLM.int8() (L8): Dynamically chooses how precisely to represent semantics internally to improve memory efficiency.
   • KV Cache (KV): Stores attention keys and values generated during sequence generation so they can be reused in later inference steps.
   • RAGCache (RC): A multilevel dynamic caching system for RAG that caches intermediate knowledge states and overlaps retrieval with inference.
8. Quadrant VIII: Semantic Memory
   • MemoryLLM (ML): A mechanism for editing a model directly with new knowledge so it integrates with old knowledge.
   • WISE (Wi): Retains pretrained knowledge, but keeps an editable side memory model that the LLM can choose when relevant.

They categorize memory into 8 categories:

1. Conversational Memory: Short-term conversational memory. This is what most people think of when they think of chatbots: chatbots are (now) pretty good at keeping up with the gist of a conversation and referring to things said earlier.
2. Episodic Memory: Long-term event-based memory. This includes past conversations, personalization, newly established facts, and anything time-related. Normally, this is where RAG systems live.
3. Contextual Memory: Loosely, the context of a current conversation. According to the paper, very little research has been done in this area; conversational context is generally tracked with words, not in model weights.
4. Identity Memory: A fuzzy mental model of who the AI is, who it interacts with, and facts about those people. Research here is focused around potentially modifying the AI weights themselves to encode new memories, beyond just fine-tuning or LoRA.
5. Working Memory: Steps in reasoning that help AI solve a current complex problem. This is basically chain of thought reasoning, where AI talks to itself to work through complexity.
6. Procedural Memory: Reflections of previously attempted problems that help AI to do better on future problems. This is like, reasoning after the reasoning, the idea being that AI can learn how to do logic better.
7. Attention Memory: A working space for what parts of the conversation are relevant. Most efforts here are performance-based, tricks used to optimize the internal working memory of the algorithm so it is cheaper and faster to run.
8. Semantic Memory: A fuzzy mental model of all the world's facts and how they relate to each other. For all intents and purposes, this is what an AI model is. There is some research in making AI model weights dynamic so it can learn.

In my opinion, there are two reasons, the first practical and the second philosophical.

Proper encoding and retrieval of memories helps ground the truth. That AI hallucinates falsehoods or bad code is a very valid criticism. Besides simply misinforming people, an AI with a wrong model of the world can make some horribly bad decisions, in biology and medicine for example. All the research being poured into memory are indirectly tackling all the ways in which AI can hallucinate.

Our sense of self is likely intertwined with our memories. Part of who we are is rooted in the connected stream of events that we have witnessed and remember. LLMs don't have that, they don't normally have episodic memories, or if they do it behaves very different from ours. But what if we erode that difference? If we're worried about the ethics of AI consciousness, then monitoring strides in AI memory is vastly important.

Imagine that you have a database of facts. In this database, you have:

• The birthdays of 10,000 individuals.
• Documents for how these individuals are related to each other (family, friends, etc).
• Trips that these individuals have taken, or will take in the future.

It's a lot of data, but now we ask the AI:

Is my sister's birthday before or after my trip?

If you were building an AI system, how would you solve this problem?

You can't just use naive RAG; a semantic search cannot isolate the specifically needed facts, resulting in forcing the AI to "find a needle in a haystack" and reason over hundreds of thousands of permutations.

I'll try to make a follow-up post this week, since while writing this article I realized RAG probably does work for this exact problem, just not regular RAG. In other words, you probably don't need any of the alternative memory methods brought up in the AI memory survey discussed above.