Memory and State

Memory in a language model starts as pure context: whatever fits in the active window. That is enough for a single chatbot turn, but real AI products operate across sessions, accumulate knowledge about users and tasks, and need to recall information that was too large or too old to stay in the window. Memory architecture is the discipline of deciding which information lives where, how it gets there, and how the system pulls the right piece at the right time without injecting noise or returning stale facts.

Memory as a System

The clearest way to reason about AI memory is a three-tier hierarchy. Each tier trades latency for capacity and persistence.

Each tier serves a different purpose. L1 is where reasoning happens: the model sees everything in the window simultaneously, at full attention resolution. L2 is where relevant history gets surfaced: vector search pulls in past interactions that are semantically similar to the current query. L3 is where stable facts live: user preferences, confirmed settings, and entity relationships that need exact retrieval rather than approximate match.

The architectural discipline is deciding what belongs in each tier. Store a confirmed account balance as an episodic embedding and retrieval becomes approximate; one day the wrong value surfaces. Store it in L3 as a key-value record and it is always exact.

Working Memory: The Context Window

L1 is effectively free on a per-retrieval basis, but it has two hard constraints. First, it is bounded: even a 2M-token window has a limit. Second, it resets: when the session ends, everything in it disappears unless explicitly persisted to L2 or L3.

The practical problem is that context fills up. A 20-step agent retrieving 2,000 tokens per tool call will exhaust a 128K window before finishing the task. Two standard techniques address this:

Sliding window: keep the N most recent messages and drop older ones. Zero cost, high fidelity for recent turns, but discarded information is gone permanently. Good for tasks where only recent state matters, like a step-by-step form fill.

Summarization: compress older exchanges into a shorter digest and drop the raw messages. Preserves semantic content for longer than a sliding window, but adds a model call and latency. Good for tasks where the full arc matters, like a multi-day research project.

A practical hybrid: keep the original goal verbatim at the top of the context (never summarize it), followed by the most recent compressed summary, followed by the K most recent raw turns. Regenerate the summary every M steps. The original goal anchors the agent even after the context has been compressed several times.

Prefix Caching and Prompt Structure

Modern inference servers (vLLM, Anthropic, OpenAI) support prefix caching: the server holds the KV cache for the static portion of a prompt in GPU memory across requests. If your 4,000-token system prompt and 50 tool schemas always appear at the front of every message, each call after the first one pays only for the new tokens.

The design rule that follows: place immutable instructions at the front of every prompt. Never put dynamic content (timestamps, user names, session variables) in the prefix. Any change to the prefix invalidates the cache for every user on that server.

Long-Term Memory: Episodic and Semantic

When a session ends, worth-keeping information must move out of L1 before the window resets. The right destination depends on what kind of information it is.

Episodic Memory

Episodic memory stores what happened: past interactions, task trajectories, and session summaries. The standard implementation is a vector database. At the end of a session, an extraction step compresses the conversation into one or more text chunks, embeds them, and stores them alongside metadata (user ID, timestamp, session ID, confidence score).

At the start of a future session, the system embeds the current query, searches the vector store for the K most similar past chunks, and injects the results into L1. This is the same retrieval pipeline as RAG applied to conversational history instead of documents.

Semantic Memory

Semantic memory stores what is true: facts about entities, confirmed user preferences, and attributes. These require exact retrieval, not fuzzy search. A vector database is the wrong tool here. Storing "user prefers dark mode" as an embedding and retrieving it by similarity risks returning the wrong user's preference if two embeddings happen to be close. Use structured storage keyed by user ID or entity ID.

A production memory system combines both: vector search for contextual personalization and episodic recall, plus a relational or graph store for confirmed ground-truth facts.

The Vector Write and Read Path

The write path runs after sessions end. New information arrives, an extraction step identifies worth-remembering content, an embedding model converts it to a vector, and the vector is stored with metadata.

The read path runs at the start of each session or each agent turn. A query arrives, the embedding model converts it to a vector, approximate nearest-neighbor search returns the top-K candidates, each candidate's similarity score is checked against a threshold, and only candidates that pass are injected into L1.

Retrieval Threshold Gotchas

The similarity threshold is one of the most consequential hyperparameters in a memory system, and it is almost always misconfigured at first.

Set it too low (0.60 cosine similarity) and you inject vaguely related content that confuses the model: a query about "invoice payment" retrieves a memory about "salary payment" and the model blends them. Set it too high (0.98) and you get near-zero recall: nothing ever matches, and the model operates as if memory does not exist.

A starting point: 0.80 to 0.85 cosine similarity for general conversation memory, measured on a held-out evaluation set before deploying. For safety-critical or factual domains, tighten to 0.90 or above. Log every retrieval event with its score. That log becomes the labeled dataset you need to tune the threshold properly.

Semantic Caching

Semantic caching applies the same vector-search idea to LLM responses: instead of retrieving past memories, it retrieves past answers. If an incoming query is semantically close enough to a previously answered query, return the cached answer and skip the LLM call entirely.

The pipeline:

1. Embed the incoming query
2. Search the cache (a vector store indexed by query embeddings)
3. If the similarity score exceeds the cache threshold, return the cached response
4. If not, call the LLM, store the result under the query embedding, and return the new response

At high volume this cuts costs by 30 to 70 percent and drops latency from seconds to milliseconds for cache hits.

Threshold Tradeoffs in Caching

The threshold logic in caching mirrors the memory retrieval case but with different failure modes. A loose threshold (0.80) gives a high hit rate but risks semantic drift: a query about "how do I reset my password" might return the cached answer for "how do I change my username" if both mention account settings. A tight threshold (0.98) gives near-zero drift but also near-zero cache benefit.

The right threshold depends on domain. Factual, repeatable queries (policy lookups, FAQ answers, math problems with fixed inputs) tolerate a threshold around 0.90 because semantically similar questions genuinely share the same answer. Creative or personalized queries (write me a poem, summarize this article) should not use semantic caching at all, because the correct answer depends on the exact input rather than just the intent.

Conversation State Management

Memory stores content. State stores the control structure of a running agent: what step it is on, what it has decided, and what remains open. These are different concerns and should be managed separately.

A well-typed state object might look like this:

Copyclass AgentState(TypedDict):
    messages: list[Message]          # L1 working memory
    current_goal: str                # Original task, never overwritten
    plan: list[str]                  # Decomposed steps
    completed_steps: list[str]       # Append-only log
    tool_results: dict[str, Any]     # Latest outputs, pruned after use
    session_id: str                  # Foreign key into persistent store
    iteration_count: int

The current_goal field is kept verbatim from the start and never modified. This anchors the agent even after many steps of intermediate context have been summarized or dropped. The completed_steps list is append-only, which prevents the state from going backward silently.

Checkpointing and Resumability

For agents that run for minutes or longer, state must be persisted after each step. If the server crashes at step 14 of 20, a checkpointed agent resumes from step 14 rather than starting over. The mechanics are straightforward: serialize the state object to a database row keyed by session ID, and load it at the start of each step.

Checkpointing also enables human intervention. A reviewer can inspect the persisted state mid-run, correct a wrong assumption, and let the agent continue from the edited point without discarding all prior work. This is especially useful for catching wrong plans early in a long task.

Failure Modes

Stale Memory

A user changes a preference. The new value is written to L3. But an older embedding in L2 still reflects the previous preference and happens to score slightly higher on the next retrieval. The model uses the stale value.

Mitigation: treat L3 as the authoritative source for any fact that can change. When a fact updates in L3, invalidate or re-embed the related L2 entries. Give L2 entries a TTL calibrated to how quickly the underlying information changes.

Memory Contradiction

Two memories say opposite things. "User is on a free plan" (stored three months ago) and "User upgraded to pro" (stored last week) are both in L2. Both are retrieved because both score above the threshold for a billing query. The model now has a contradiction to resolve.

Mitigation: facts that change over time belong in L3 structured storage where an update simply overwrites the prior value. Attach timestamps and confidence scores to all L2 entries. If two entries contradict each other, prefer the more recent one and log the conflict. For the model's context, present one entry rather than both: showing a contradiction tends to produce hedged, unhelpful answers.

Unbounded Growth

A memory store with no expiry policy degrades over months. Low-quality old memories accumulate and compete with high-quality recent ones. Retrieval precision drops as the index grows.

Mitigation: implement a decay function. Access-based decay downranks memories that are never retrieved. Recency-based decay moves entries older than N days to cold storage unless they have been accessed recently. Consolidation-based decay periodically merges many low-level memories on the same topic into one high-quality summary node, shrinking the index while preserving the substance.

A Worked Example: Multi-Session Customer Support Agent

A user opens a support ticket about a billing discrepancy. This is their third conversation this month.

Session start: the agent embeds the user's first message and searches L2. It finds a memory from two weeks ago: "User reported a duplicate charge that was resolved." It also queries L3 for the user's current subscription plan and account status. The assembled L1 context is about 3,000 tokens: system prompt plus the current message plus two retrieved L2 memories plus L3 facts.

During the session: tools are called to inspect the billing system. Each result appends to L1. After 8 turns, total context approaches 18,000 tokens. The summarizer runs and compresses turns 1 through 6 into a 400-token digest, keeping turns 7 and 8 in full detail.

Session end: the extraction step identifies two conclusions worth storing: "Agent confirmed duplicate charge was a system error, refund issued on [date]" and "User satisfied with resolution, no escalation needed." Both are embedded and written to L2 with metadata. The refund amount and date are written to L3 as structured key-value facts.

Next session: if the user opens a new ticket about billing, L2 surfaces the prior billing history and L3 supplies the exact refund record. The agent knows the context without the user repeating themselves.

Interview angle