ZICQ
中 Log in / Sign up
Newsroom Agentic #LLM Agents #Long-Term Memory #Memory Filtering #Resolver Errors #Cost-Sensitive Decision

When Should LLM Agents Forget? Memory Filtering Under Query-Critical Resolver Errors

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 1 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:An arXiv preprint reframes stale-fact deletion in LLM memory as an asymmetric decision problem rather than a default cleanup rule. When a resolver falsely deletes a fact that is critical to the current query, downstream accuracy degrades significantly more than when a stale fact is incorrectly kept. Across three models, hard deletion changes from gains of +0.186 to +0.230 at perfect resolution to losses of -0.176 to -0.181 at the lowest tested resolver reliability; paired targeted interventions


This study reframes stale-fact deletion in LLM long-term memory as an asymmetric decision problem rather than a default cleanup action. The central result is direct: when a resolver falsely deletes a fact that is critical to the current query, downstream accuracy degrades significantly more than when a stale fact is incorrectly kept. At the same time, never deleting is not risk-free because incorrect labels may still steer generation. The paper proposes a deployment-matched gate instead of a universal forgetting threshold.

Core finding: the value of deletion reverses with resolver reliability

A hard-filtering pipeline typically labels memory facts as stale, removes the labeled items, and sends only the remaining context to the model. The problem is that resolver labels are imperfect, while the importance of a fact depends on the active query. A fact may be globally outdated but still necessary for answering the current question.

The researchers ran a controlled sensitivity intervention across three models. With perfect resolution, hard deletion produced gains ranging from +0.186 to +0.230. At the lowest tested resolver reliability, those gains reversed into losses of -0.176 to -0.181. Stale-information cleanup is therefore not inherently beneficial: as resolver quality degrades, the deletion operation itself becomes a primary source of risk.

Paired targeted interventions compared false-drop and false-keep errors. False-drop occurs when a query-critical fact is labeled stale and removed; false-keep occurs when a stale fact is retained. The abstract reports that false-drop causes significantly greater degradation. Deletion creates irreversible information loss, while an incorrect retention mainly adds context noise that the model may still overcome using other evidence.

Nevertheless, the non-deleting baseline can also fail. If incorrect labels continue to influence ranking, retrieval instructions, or generation guidance, the model may rely on a fact that should have been suppressed. Retaining everything is therefore not a zero-risk solution. The appropriate framing is cost-sensitive deletion rather than unconditional deletion.

Mechanism: query-criticality is different from global staleness

Let q denote the current query, M the memory set, and D the set removed by the resolver. Hard filtering removes D before context construction, creating an irreversible information bottleneck. A fact can be stale as a general assertion while remaining critical to q. Staleness is an intrinsic property of the fact; query-criticality is a conditional relationship between that fact and a particular request.

One engineering approximation is:

ΔU ≈ benefit of stale-fact removal − P(false-drop ∩ query-critical) × C_FD − P(false-keep) × C_FK

Here, C_FD is the cost of deleting a query-critical fact, while C_FK is the cost of retaining or acting on a stale fact. The paired experiments support C_FD being larger than C_FK in the evaluated setting. The crossover for deletion depends on false-drop probability, the proportion of query-critical facts, and the downstream costs of the two error types—not on resolver accuracy alone.

This is why the paper does not provide a universal threshold. Model, task, prompt, context length, and fact-update frequency can all change C_FD and C_FK. Moreover, identifying a global accuracy crossover requires gold labels. The crossover is a model- and task-specific diagnostic, not a production cutoff that can be copied across systems.

What the experiments show

The paper uses a frozen 102/102 calibration/test split and compares three strategies:

| Strategy | Accuracy | Engineering interpretation | |---|---:|---| | Always hard-drop | 0.641 | Lower context usage but full exposure to false-drop risk | | Deployment-matched gate | 0.655 | Chooses retention or deletion using deployment-matched evidence | | Test-informed oracle selection | 0.662 | Uses test information and therefore serves only as an upper-bound diagnostic |

The deployment-matched gate is 0.014, or 1.4 percentage points, above always hard-drop and remains 0.007 below the oracle. Because the abstract provides no confidence intervals, significance tests, or per-task breakdowns, these values alone cannot establish that the gate is uniformly and statistically superior. The frozen split, controlled reliability sweep, and paired error interventions nevertheless provide more explanatory evidence than a single average score.

Engineering trade-offs: no option is free

| Strategy | Main benefit | Main cost or risk | |---|---|---| | Always hard-drop | Shorter context with lower token, latency, and distraction costs | One false deletion can remove irreplaceable evidence | | Always retain | No physical false deletion and complete retrieval paths | Context growth, stale guidance, higher inference cost, and potential compliance exposure | | Soft filtering or quarantine | Query-aware adjustment with rollback potential | More routing, ranking, and audit complexity | | Deployment-matched gate | Calibrates error costs to the model, task, and traffic | Requires matched data and periodic recalibration under distribution drift |

A critical implementation boundary is often missed: relevance filtering and compliance deletion must remain separate. Sensitive information subject to a user deletion request cannot survive merely because it may still be useful to some query. Conversely, a deletion performed for privacy or lifecycle policy should not be conflated with a resolver's stale-fact label. The two paths require different permissions, audit semantics, and irreversibility guarantees.

Evaluation limits and replication risks

The abstract does not disclose the names and versions of the three models, dataset and task composition, resolver implementation, label-generation procedure, calibration distribution, confidence intervals, prompt and decoding settings, or code and license availability. Consequently, several claims still require full-paper inspection and independent replication:

  • Whether the 0.655 versus 0.641 gap remains stable across models, tasks, and traffic distributions;
  • Which features the deployment-matched gate uses and whether it creates a new source of overfitting;
  • Whether the asymmetry between false-drop and false-keep persists across memory sizes, context lengths, and degrees of fact conflict;
  • How factuality, abstention, retrieval recall, token cost, and tail latency change alongside aggregate accuracy.

Freezing the calibration and test split reduces repeated tuning on the same test set, but it does not remove mismatch between calibration data and production traffic. A gate tied to a model version, user population, or task template should therefore be monitored for drift and recalibrated in the same way as any production threshold.

Deployment blueprint for developers

  1. Separate storage, lifecycle deletion, and relevance filtering. Manage raw facts, expiry policy, compliance erasure, and context injection through independent control planes instead of overloading one stale label.
  2. Replace global staleness with query-conditioned scoring. Combine resolver confidence with an estimate of the fact's potential relevance to the current query; a globally stale label should not automatically trigger irreversible removal of a query-critical fact.
  3. Prefer reversible quarantine. Archive, version, or downrank medium-confidence facts instead of deleting them immediately, preserving a short rollback window.
  4. Estimate both error costs on matched data. Replay resolver decisions using representative queries and measure how false-drop and false-keep affect final answers. Without matched validation, the paper recommends retaining facts rather than applying hard deletion.
  5. Create an auditable decision log. Record fact versions, resolver labels and confidence, query-criticality scores, final actions, and model versions so failures can be replayed without rerunning the model.
  6. Run in shadow mode, then canary. Execute the new gate alongside the legacy policy without applying deletions, compare candidate differences, and enable destructive actions only after the benefit is stable.

A practical runtime flow is:

write fact and version → resolver emits state and confidence → score query-criticality → gate selects retain, quarantine, or delete → construct context → log decision and outcome

Bottom line

The paper's main contribution is turning the question of whether an agent should forget into a measurable error-cost problem. Hard deletion can improve accuracy when resolver labels are reliable, but as reliability falls, query-critical false drops become more damaging than the benefit of cleanup. The safer production pattern is neither permanent forgetting nor permanent retention. It is reversible, auditable gating calibrated against query-critical false-drop risk on matched data. When that evidence is unavailable, retaining facts while controlling their exposure is preferable to applying an unvalidated global staleness threshold.


Source: ArXiv AI (cs.AI) (2026-11-17)

— END —

Tags: #LLM Agents #Long-Term Memory #Memory Filtering #Resolver Errors #Cost-Sensitive Decision

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…