How does context compaction work?

TL;DR

• Context compaction reduces a long conversational or document context into a shorter representation that keeps task-relevant information.
• Common techniques: selective retrieval, extractive selection, abstractive summarization, and learned encoders that produce compact embeddings.
• Compaction trades raw fidelity for token efficiency and faster inference; you must measure reconstruction or task performance loss.

Key tradeoffs: fidelity versus compactness, latency versus accuracy, and simplicity versus maintainability.

In this question, we will learn how context compaction works and why agents use it to manage long histories and large memories. We will keep things practical and show concrete steps you can implement in an agent pipeline.

We will cover the following:

• The intuition
• How it actually works
• Common algorithms and a comparison table
• Design patterns and when to compact
• Tradeoffs and failure modes

Direct answer: Context compaction means selecting or transforming a large context into a smaller representation that preserves task-relevant facts while fitting model token limits. We do this with scoring and selection heuristics, extractive or abstractive summarization, and sometimes learned compression models, accepting some loss of detail to keep inference fast and within budget.

The intuition (an analogy that makes it click)

Imagine you are preparing notes for a meeting where you can only bring one sheet of paper. You would not copy the entire chat log. Instead you highlight the decisions, open action items, and a tiny timeline of who said what. Context compaction is the same. We condense the history into a short sheet that helps the model act correctly, while leaving out low-value chatter.

How it actually works (the real mechanics, with one concrete worked example)

At a high level the pipeline has steps: chunking, scoring, selecting, and optionally compressing into an abstraction. Here is a concrete example using a chat history of $n$ tokens that we want to reduce to at most $m$ tokens.

1. Chunk the history into segments of roughly equal token length, e.g., messages or paragraphs.
2. Score each chunk for relevance to the current task using heuristics or a learned scorer like cosine similarity on embeddings, or a small model predicting relevance.
3. Select top chunks until the token budget is nearly filled, or use an abstractive summarizer on the union of top chunks to produce a shorter synthesis.
4. Store the compact representation in the agent context or memory store.

A simple compression ratio is:

$$r = \frac{m}{n}$$

where $r$ is the fraction of tokens we keep. If the original $n=12{,}000$ and we need $m=1{,}000$ then $r\approx 0.083$.

Worked numeric example: suppose a chat broken into 6 chunks with token counts and a relevance score between 0 and 1.

If the token budget for compacted context is $m=1500$, a greedy selection by score per token might pick C1, C2, C5 (250+400+1200 = 1850 which is too large). Instead we could: take C1 and C2 in full (650 tokens), then request a 400-token abstractive summary of C5 and a 450-token summary of the high-value parts of C6 to hit $m=1500$. That keeps the most relevant facts while fitting the budget.

Common algorithms and a comparison table

We often choose among several standard methods. Here is a compact comparison:

Choose based on whether you need verbatim evidence or only task cues.

Design patterns and when to compact

• Recent-window plus summary: keep the last $k$ messages in full and a rolling summary of older material. This is a balanced default.
• Memory layering: store short-term facts in a fast context, long-term knowledge in a compact store with stronger compression and retrieval.
• Recompute vs incremental update: full recompression gives fresher summaries but costs compute; incremental updates append new facts and update summaries to save CPU.

When to compact: compact when total tokens exceed model limits or when latency and cost matter. Also compact periodically to keep long-running agent sessions usable.

Tradeoffs and failure modes

Failure modes to watch for:

• Lost preconditions: compacted context may omit constraints the agent needs.
• Summarizer hallucinations: fabricated facts can mislead downstream reasoning.
• Overfitting the scorer: a poor relevance model can filter out essential data.

Questions the interviewer might ask:

Some follow-up questions you might get:

How do you choose between extractive and abstractive compaction? Extractive is safer when you need exact quotes or audit trails. Abstractive works when you prioritize brevity, but you should validate fidelity when using it.

How do you measure whether compaction is hurting performance? Run task-specific metrics such as accuracy, success rate, or information recall. Compare runs with and without compaction and measure delta in those metrics.

Can embeddings replace text summaries? Embeddings are great for retrieval and scoring, but they do not provide readable evidence. Use embeddings to select candidates and then summarize the text if you need a human-readable context.

How often should you re-run summarization for a long session? It depends on update rate and cost. A common pattern is after each significant event or every N messages, or when the token budget threshold is exceeded.

How do you avoid summarizer hallucination? Prefer extractive or conservative summarizers, add verification steps that cross-check summary facts against the source, and keep a retrieval path to original chunks.

Some things to note:

• Always keep the ability to fetch original text for critical decisions.
• Use hybrid strategies: recent full context plus compressed long-term memory.
• Evaluate on the actual downstream task, not just summary quality metrics.

What the interviewer is really testing

They want to see that you understand the practical limits of models and that you can design pipelines that trade tokens, compute, and fidelity effectively. They also expect you to reason about failure modes like hallucinations and to propose measurable ways to validate compaction choices. Finally, they want concrete patterns you can implement in an agent, not just abstract theory.