“The agent remembers me” sounds like a feature until it recalls a sensitive detail in the wrong conversation, treats an outdated preference as permanent or cannot explain where a fact came from. Memory is not a larger prompt history. It is a product data system with scope, ownership, confidence, retention and deletion behavior. Decide what deserves persistence before choosing a vector store.
| Information | Default treatment | Required control |
|---|---|---|
| Current task details | Short-lived thread state | Retention window and clear/reset action |
| “Use concise answers” | Durable preference only with clear user benefit | Visible, editable and removable |
| Project deadline | Store with provenance and expiry | Refresh or mark stale automatically |
| Health, financial or identity detail | Do not store by default | Specific purpose, lawful basis and strict access if essential |
| Password, token or recovery code | Never put in agent memory | Use a dedicated secrets manager outside model context |
| Inferred personality or sensitive trait | Do not convert model speculation into fact | Require explicit confirmation and necessity |
Conversation continuity and durable personalization are different needs. A thread checkpoint can preserve a workflow so it resumes after interruption; a user-scoped store can retain a small set of preferences across sessions. Do not copy the entire chat transcript into a “memory” table by default. Summaries can still contain sensitive information, so they need the same access, expiry and deletion controls as source messages.
Partition by user or organization at the storage layer. Retrieval should enforce that scope before data reaches the model. A prompt instruction is not an access-control system. Audit reads and writes for sensitive categories without logging the sensitive values themselves.
For each stored item, record a type, subject/scope, source, creation time, confidence, last-confirmed time and expiry. Show people what the system remembers in understandable language, not raw embeddings. Let them correct, remove or disable memory. If an agent proposes a memory based on inference, ask for confirmation when the impact justifies it; otherwise keep it temporary.
Deletion must cover primary records, derived summaries, vector indexes, caches and backups according to the stated retention policy. Mark an item for deletion immediately so it is no longer retrieved while asynchronous cleanup runs. Define what happens to audit events and legal holds, and communicate the boundary instead of promising instant erasure from every backup. Periodically test deletion with a search for the deleted identifier and paraphrased content.
Expiry is not deletion if stale data remains queryable. Use TTL or scheduled cleanup, but also remove the item from embeddings and indexes. Reconfirm facts that can become wrong, such as role, employer, project status or location. On low confidence or contradictory sources, ask rather than silently overwriting.
Data minimization is stronger than collecting everything and hoping access controls are enough. Store a short preference instead of a transcript, a pointer instead of a document copy, and a task-scoped note instead of an enduring profile. Never store credentials in model-readable memory; use an application secret store and narrowly scoped tool credentials.
Memory can become prompt-injection material too. Treat recalled text as untrusted input, delimit it, and do not let it change tool permissions or system policy. Test whether one user can retrieve another user's memory, whether deleted facts reappear and whether crafted content can plant durable instructions.
Track whether memory actually reduces repeated questions or improves task completion, alongside stale-memory corrections, deletion latency, cross-scope access failures and user opt-outs. Compare behavior with memory enabled and disabled. If the benefit is marginal and the privacy surface grows, do not persist the information.
Good agent memory is small, scoped, explainable and forgettable. Keep task state temporary, persist only useful user-approved facts, attach provenance and expiry, and make deletion reach every derived store. Secrets and speculative sensitive traits do not belong in conversational memory. Personalization should earn persistence through clear value.