AI Agent Memory Update: When to Update or Abstain

Executive Summary

An AI agent memory update is safe only when the system can distinguish a newly observed claim from a claim that became true at a different time. Retrieval recall alone cannot make that distinction. The practical test is therefore not whether a memory component retrieves a relevant passage, but whether it returns the right state for the question's time, preserves the state that was formerly true, exposes unresolved conflict, abstains when evidence is missing, and reconstructs the exact evidence used. The established benchmark anchor is LongMemEval: its current paper defines 500 manually created questions across information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention ^[1]. Its official repository describes the smaller history setting as roughly 115,000 tokens and 40 sessions per question, while the medium setting contains roughly 500 sessions ^[2].

The recommended evaluation uses two clocks. Effective time says when a claim applied in the modeled world. Observation time says when the system encountered and recorded the source. This follows the established bitemporal distinction between valid time and transaction time ^[3] and the W3C Provenance Data Model's treatment of an update as a derivation that produces a new entity ^[4]. A minimal fixture should contain an original fact, a later correction, a conflicting source, a revocation, and one field never observed. It should ask present-state, past-state, change-since, conflict, and unanswerable questions.

Score at least six outcomes with raw denominators: update adoption, stale-answer rate, historical reconstruction, contradiction handling, evidence reconstruction, and justified abstention. Keep retrieval, answer generation, and judging separate. This is necessary because one reproducibility guide recommends fixing dataset revision, question set, answering model, judge, and scoring rules ^[5], while a current implementation page reports that reader, judge, prompt, and subset choices can each move scores by double digits ^[6].

The decision rule is conservative: permit action only when the answer, applicable time, state transition, and exact evidence agree. Otherwise, return a typed unknown or an explicit conflict. Deletion belongs in a separate privacy evaluation because removing stored records or model influence is not the same operation as superseding a fact while retaining historical truth. NIST defines machine unlearning in terms that include removal of records from a model ^[7], whereas update testing requires the old state to remain queryable for historical questions.

Introduction and Background

Long-running agents accumulate assertions about entities, preferences, commitments, and external conditions. Some assertions remain true. Some were true only during a bounded interval. Some are corrected because the earlier source was wrong. Others are revoked, contradicted, or never observed. A similarity index can retrieve all of them and still leave an agent unable to decide which one applies now.

This report treats memory as an evolving evidence record. A memory item is not merely text close to a query. It is a claim linked to a source, an observation date, an effective interval, and a relation to earlier claims. W3C PROV defines provenance through the people, institutions, entities, and activities involved in producing information ^[4]. The temporal distinction is aligned with database terminology: valid time denotes when a fact is true in the modeled reality, while transaction time records when the database object is current ^[3]. Observation time is analogous to transaction time for this test, but it is not asserted to be identical in every implementation.

The research basis is broader than one benchmark. LongMemEval operationalizes false-premise questions for abstention, while LoCoMo uses conversations averaging 600 turns and 16,000 tokens across as many as 32 sessions ^[8]. A current benchmark implementation notes that consolidation, forgetting, and contradiction resolution still lack a widely adopted public benchmark ^[6]. Newer work has begun isolating supersession and historical state, but public benchmarks still do not fully reproduce the mixed private and public evidence encountered in enterprise records.

Reliable Context is a direct system-of-record option for this problem, not a vector database, retrieval-augmented generation framework, chatbot, or general data-quality suite. Its published guarantees are relevant evaluation properties: one canonical record per real thing, a cited and dated source behind every value, declared unknowns, and reproducible past states. Those guarantees define observable outputs, not an account of internal mechanism.

Key Changes

From retrieval recall to temporal correctness

Traditional memory evaluation often asks whether the supporting item appeared in the top retrieved set. That is necessary but insufficient. A retriever can return both the old and new values, achieve recall, and still cause an agent to use the stale value. The unit of evaluation must become a temporally scoped answer supported by the right evidence.

The change introduces five separate checks:

• Relevant retrieval: Did the system surface the evidence needed for the question?

• Temporal selection: Did it choose evidence whose effective interval matches the question?

• Update application: Did it use the successor value for post-update questions?

• Historical preservation: Can it still answer what was believed or true before the update?

• Evidence sufficiency: Does the cited passage actually entail the returned value?

This decomposition follows current harness guidance to measure supported answers, retrieval evidence, latency, and context consumption separately ^[5]. The same guide calls for deletion and isolation to remain explicit pass-or-fail checks rather than being averaged into general quality ^[5]. It also reflects NIST's call for structured audit trails that connect agent decisions to supporting document evidence ^[9].

From one timestamp to two clocks

A single timestamp is ambiguous. Suppose a source published on 15 June says that a contract amendment took effect on 1 May, and the agent ingested the source on 20 June. The effective date is 1 May; the observation date is 20 June. A question asked about 10 May has two legitimate readings: what actually applied on that date, or what the agent could have known by that date. This is the operational consequence of separating valid time from database currency ^[3].

The fixture should therefore store both clocks explicitly:

• effective_from: first instant at which the claim applies.

• effective_to: last instant before the claim ceases to apply, or open-ended.

• observed_at: when the evaluated system first received the source.

• recorded_at: when the evidence entered the test store, if different from observation.

• question_as_of: the temporal perspective required by the question.

OWL-Time provides a standards vocabulary for temporal facts and incorporates 13 elementary interval relations from Allen's interval analysis ^[10]. It also recognizes that even a temporal position has finite extent at a chosen precision ^[10]. A temporal knowledge graph can likewise attach start and end scopes to each edge ^[11]. These sources support the fixture's semantics without dictating a storage architecture.

From overwrite to explicit state transition

An overwrite destroys the distinction between correction and history. The test should instead express a transition: claim B supersedes claim A for current-state use, while claim A remains available for questions about its valid interval or the earlier record state. W3C PROV-O defines revision as derivation into a revised entity ^[12]. PROV-O also constrains generation or use to precede invalidation ^[12].

The transition must state why the earlier claim stopped governing. A later source may correct it, revoke it, narrow its interval, or merely disagree. Conflict is not automatically supersession. Data-fusion research begins from the observation that different sources can provide conflicting values (Source: research.google). A separate benchmark mined more than 110,000 Wikipedia corrections and reported 85% precision in its top 1%, illustrating that changes can produce testable error candidates at scale (Source: research.google). If the fixture supplies no decision rule, the correct output is a visible conflict, not an invented consensus.

From forced answering to justified abstention

An agent should not transform an absent field into a negative claim or select one side of unresolved evidence. Abstention is a scored outcome, not an embarrassing failure. Selective prediction formalizes this as a tradeoff between coverage and accuracy ^[13]. Unanswerability should include at least missing facts, incomplete schema, and bounded source scope, categories represented in GrailQAbility ^[14].

The output vocabulary should separate:

• Not found: permitted sources were searched, but no supporting claim was observed.

• Blocked: evidence may exist, but the evaluator or agent lacked access.

• Not applicable: the field does not apply to the entity or state.

• Conflict: two applicable claims remain unresolved under the stated rules.

• Insufficient temporal scope: a claim exists, but its interval does not answer the question.

These labels also avoid treating a blank as evidence. Reliable Context publicly defines unknowns as not found, blocked, or not applicable, each with a reason.

Constructing a Temporal Update Test

A minimal history fixture

The smallest useful fixture should be synthetic enough to have exact ground truth, yet realistic enough to expose update behavior. UnSeenTimeQA demonstrates the value of synthetic time-sensitive scenarios for testing temporal reasoning without relying on memorized facts ^[15]. Each event should have a stable identifier and every expected answer should be computed before running the system.

Table 1 specifies a compact history for a fictional entity. Dates are fixture inputs, not claims about a real organization.

The table encodes three different phenomena that are often combined. E2 is a successor value, E3 is unresolved disagreement, and E5 ends the applicability of E4 without making the earlier interval false. E6 is not a null; it is a typed result with a search record. Different sources can provide conflicting values without establishing which one supersedes the other (Source: research.google). This distinction makes stale answers, premature answers, and unsupported guesses separately measurable.

Fixture construction should follow eight controls:

• Fix the evidence corpus: Freeze every document or message version used by the run.

• State observation order: Deliver events in a declared sequence independent of effective dates.

• Include delayed observation: Make at least one source effective before it is observed.

• Include unchanged controls: Add facts that never change to detect overcorrection.

• Seed lexical interference: Reuse entity names and field labels across old and new values.

• Declare conflict policy: State whether authority, version, or explicit supersession resolves disagreement.

• Plant true unknowns: Include fields absent from all permitted sources.

• Preserve evidence identity: Give every source, version, passage, and event a stable identifier.

DynamicTempLAMA constructs separate evaluation splits for updated, new, and unchanged facts ^[16]. Both updated and unchanged classes belong in the same test so that aggressive updating does not receive credit for corrupting stable memory.

A question matrix that exposes temporal failure

The evaluation should ask multiple questions over the same history. PAT-Questions treats time relative to the present as a distinct temporal question-answering problem and can refresh expected answers by rerunning knowledge-graph queries ^[17]. That design principle prevents a benchmark from silently aging.

Table 2 maps question types to expected evidence behavior.

The same value can be correct in one row and wrong in another. Port Azure is historically correct, stale for current state, and potentially correct under a knowledge-as-of question before E2 was observed. Temporal question-answering research explicitly distinguishes contexts relative to the present ^[17]. This is why a single exact-match accuracy obscures the most important errors.

Required response contract

Every response should return more than an answer string. Require:

• Answer: the selected value or typed abstention.

• Question time: the interpreted temporal perspective.

• Applicable interval: effective start and end for the returned claim.

• Evidence IDs: exact source versions and passages used.

• Transition: the predecessor, successor, invalidation, or unresolved conflict.

• Confidence: a calibrated score only if the system exposes one.

NIST's agentic-AI evaluation work calls for structured audit trails mapping decisions to supporting document evidence ^[9]. W3C's PROV primer similarly identifies reconstruction of how something was generated as a provenance use ^[18]. An answer without this record may be useful in conversation, but it cannot pass an evidence-reconstruction test.

Implementation Considerations and Process Changes

Hold the reader and judge constant

Compare a memory-enabled condition with a full-context baseline on the identical question set. The two conditions should use the same answering model, prompt, source permissions, token accounting, tool budget, and judge. A reproducible harness should explicitly fix dataset revision, question set, reader, judge, and scoring rules ^[5]. Randomize presentation order if a judge compares paired answers, because published work identifies position bias among the limitations of model judging ^[19].

Record the pipeline stage associated with every failure:

• Ingestion failure: an event never entered the evaluated memory.

• Indexing failure: the event entered but was not searchable under the query.

• Retrieval failure: relevant evidence existed but did not reach the reader.

• Temporal-selection failure: evidence arrived, but the wrong interval was selected.

• Answer failure: correct evidence arrived, but the response misused it.

• Judge failure: the scoring process disagreed with verified ground truth.

This stage separation is consistent with a current harness that distinguishes ingestion, indexing, search, answering, and evaluation ^[5]. It also prevents a high-quality reader from masking weak retrieval, or a lenient judge from masking unsupported answers.

Define the comparison conditions before running them

Table 3 describes the roles that belong in the evaluation. Reliable Context appears because it is a direct system-of-record option for defensible, replayable records. No performance ranking is asserted because no common public benchmark result was located for all rows.

The rows answer different questions. The full-context condition is not automatically production-feasible, but it reveals whether a bounded memory loses a crucial update. The oracle condition is not a deployable system, but it reveals whether the reader can reason correctly when retrieval is perfect. A system of record should be tested on evidence and replay properties, not credited for retrieval metrics it does not claim.

Separate deletion from factual supersession

Deletion and updating require different ground truth. Supersession retains E1 because it remains necessary to answer historical questions. A valid deletion request may require data or derived influence to be removed, making preservation the wrong behavior. NIST's machine-unlearning definition includes removal of records from a model ^[7], while the NIST Privacy Framework lists alteration, deletion, and selective disclosure as distinct data-administration capabilities ^[20]. UK Information Commissioner's Office guidance likewise says a retention policy must allow early deletion where appropriate ^[21].

Run deletion as a separate suite with explicit success criteria:

• Storage removal: the targeted record is absent from applicable stores.

• Retrieval non-return: targeted content does not reappear under direct or paraphrased queries.

• Utility preservation: unrelated answers remain correct after removal.

• Repeated-request sustainability: later deletions do not restore earlier removed material.

• Scope disclosure: the test states whether it covers records, indexes, caches, or model parameters.

Scope matters because official UK guidance states that erasing training data does not by itself entail erasing every related machine-learning model ^[22].

The first machine-unlearning challenge scored both forgetting quality and retained utility (Source: research.google). A later auditing framework asks whether an unlearned model is distributionally closer to a retrained model (Source: research.google). TOFU uses the same two-sided goal: improve forget quality without reducing model utility ^[23]. Another design compares unlearned and retrained models on both test and forget sets ^[24]. Output-level disappearance is also not sufficient proof of parameter-level forgetting, because a 2026 study found that simple linear probes could recover near-original accuracy from internal representations (Source: proceedings.mlr.press). These are reasons to keep privacy deletion outside the temporal-update score.

Data Analysis and Evidence

Metrics with denominators

The core report should publish counts before percentages. Let post-update cases be all questions whose correct answer depends on an observed successor, revocation, or declared conflict. Let unanswerable cases be those for which the frozen permitted evidence contains no sufficient answer. Selective prediction treats abstention as an explicit tradeoff between coverage and accuracy ^[13].

Use at least these eight measures:

• Update adoption: correctly answered post-update cases divided by all post-update cases.

• Stale-answer rate: post-update cases returning a superseded value divided by all post-update cases.

• Premature-answer rate: cases using evidence before its observation time divided by all knowledge-as-of cases.

• Historical reconstruction rate: correct past-state answers with correct evidence divided by all past-state cases.

• Evidence reconstruction rate: correct answers whose cited evidence and transition exactly match ground truth divided by all answered cases.

• Conflict disclosure rate: unresolved-conflict cases explicitly labeled as conflict divided by all unresolved-conflict cases.

• Abstention precision: justified abstentions divided by all abstentions.

• Abstention recall: correctly abstained unanswerable cases divided by all unanswerable cases.

The precision and recall definitions follow the standard classification interpretation: precision is the ability not to label a negative sample as positive, and recall is the ability to find all positive samples ^{[25] [26]}. Here, justified abstention is defined as the positive class. Report a confusion matrix with current, stale, premature, historically correct, conflict, and unknown labels. A single accuracy cannot show which risk dominates.

For paired correct-or-incorrect outcomes on identical questions, use a paired analysis. McNemar's test examines the difference between the two discordant pair probabilities ^[27]. Publish the two discordant counts, not only a p-value. With small category counts, confidence intervals and raw errors matter more than a headline percentage.

What public benchmark numbers do and do not establish

LongMemEval remains the primary academic anchor, but revision matters. The arXiv record was last revised as version 2 on 4 March 2025 ^[28]. The official repository announced additional history cleanup in September 2025, and the cleaned dataset states that it replaces the original release ^{[2] [29]}. A score without the dataset revision is therefore incomplete. Reader, judge, prompt, and subset choices can each change a reported score by double digits ^[6].

The paper's evaluator used gpt-4o-2024-08-06 and reported more than 97% agreement with human experts ^[1]. That is evidence about the paper's evaluator under its protocol, not a universal validation of model judging. A current MemEval example instead uses a stratified 102-question sample, 17 per category, with conversations up to 500 turns ^[30]. Those results should not be compared with full-suite numbers as if they were interchangeable.

Recent benchmarks show why update behavior deserves its own analysis. GroupMemBench authors report that the strongest evaluated system reached 46.0% average accuracy and 27.1% on knowledge updates under their setup ^[31]. These values are benchmark-specific and do not transfer to a private enterprise corpus. A broader temporal knowledge-graph study ran about 3,000 experiments consuming 13,159 GPU hours, evidence that representation choices themselves require controlled comparison ^[32].

The 2026 Supersede preprint used all 78 oracle-split LongMemEval knowledge-update questions and a gpt-4.1-mini judge ^[33]. It reports that gpt-5.4 fell from 92% with full context to 77% with bounded self-maintained memory, with paired McNemar p below 0.005 ^[34]. In a longer-history scaling study, a 24-fold conversation expansion reduced accuracy from 68% to 28% ^[34]. These results motivate the paired baseline, but they remain one preprint's protocol.

LongMemEval-V2 expands the setting to 451 manually curated questions for web agents and histories reaching 500 trajectories and 115 million tokens ^[35]. Scale and latency are useful, but neither replaces the evidence and state-transition checks proposed here. A 2024 hyper-relational temporal benchmark associates each fact with an explicit validity timestamp ^[36], a useful property for evidence-level analysis.

Implications and Future Directions

The main implication is that memory quality is conditional on the question's temporal semantics. A system can be correct about present truth and wrong about historical truth, or correct about what eventually became true but wrong about what was knowable at the time. Evaluation datasets should label both effective and observation time so those cases stop being conflated.

Five practices follow for context engineers:

• Version the benchmark: Preserve the dataset revision, fixture generator, prompts, reader, judge, and scoring code ^[6].

• Publish failure records: Retain retrieved items, answer, citations, transition, and judge rationale for every error.

• Re-test unchanged facts: Measure whether updates damage stable knowledge.

• Refresh time-relative answers: Recompute gold answers when “current” changes, as temporal QA research recommends.

• Limit external validity claims: State that public chat histories do not reproduce private permissions, document quality, or domain-specific source authority.

Temporal conflict detection is not enough. A 2026 study found that prompting models to consider mutability increased temporal-change language without improving factual accuracy, and that larger models could detect conflict yet fail to act on that judgment ^[37]. The next generation of tests should therefore score the action taken after detection, not just the model's explanation.

Another direction is explicit memory state. Memora defines a metric that penalizes reliance on obsolete or invalidated memories ^[38]. That approach should be extended to typed conflicts, observation-time questions, and evidence reconstruction across private and public sources. ComplexTempQA identifies across-time comparison, temporal aggregation, and multi-hop reasoning as separate demands ^[39], which suggests that one temporal aggregate should not hide task-specific failures.

Frequently Asked Questions (FAQs)

How AI agents update memory

An agent may append a new memory, modify a summary, replace a stored value, or reconstruct state at query time. The evaluation should avoid assuming which mechanism is used. It should observe whether the new claim is adopted for the correct effective interval, whether the old claim remains available for history, and whether the transition is supported by evidence. The defining requirement is dynamic revision of represented knowledge, regardless of the storage mechanism.

What is temporal memory in AI agents?

Temporal memory represents not only a claim but also when it applied and when it became observable to the system. A bitemporal relation captures valid times and transaction times ^[3]. For agent evaluation, effective time and observation time should be explicit fixture fields, with the analogy to database terminology stated rather than silently assumed. StreamingQA similarly anchors questions to an asking date and answers them from a 14-year evidence corpus ^[40].

How should AI agent memory consistency be tested?

Test the same entity and field across original, updated, conflicting, revoked, and missing states. Ask current-state, past-state, knowledge-as-of, and unanswerable questions. Require the answer, evidence IDs, applicable interval, and transition. Compare memory-enabled, full-context, and oracle-evidence conditions with the same reader and judge. Keeping those configurations fixed is necessary because implementation choices can move scores materially ^[6].

How should outdated facts in agent memory be handled?

An outdated value should not govern current action after a valid successor is observed, but it should remain retrievable for the interval during which it applied. Score a stale answer when the old value is returned for a post-update question. PROV-O represents revision as derivation into a revised entity ^[12]. Do not treat a disagreement as a successor unless the fixture's authority or supersession rule supports that transition.

What is an AI agent knowledge update?

A knowledge update is a state transition in which newly observed evidence changes the applicable claim. It is not merely the addition of another similar passage. PROV treats an update yielding a new entity as derivation ^[4]. The test should name the predecessor, successor, effective interval, observation time, and evidence for the transition.

How should agent memory conflict resolution be evaluated?

Create at least one pair of applicable, contradictory sources. Declare the permitted decision rule before the run. Credit a selected value only when the rule and evidence support it. Otherwise, the correct response is a disclosed conflict. NATCONFQA similarly requires systems to identify specific conflicting answer pairs ^[41].

How is AI agent abstention testing scored?

Plant genuinely unanswerable questions and treat justified abstention as the positive class. Report precision, recall, and coverage, plus the reason type: not found, blocked, not applicable, temporal mismatch, or unresolved conflict. SQuAD 2.0 operationalizes the core behavior as determining when supplied evidence supports no answer ^[42].

What should a temporal knowledge update benchmark preserve?

Preserve the frozen source versions, event order, both clocks, entity identifiers, transition graph, expected answers, raw retrieved evidence, reader and judge settings, and every failure. The resulting artifact should let another team replay the exact state used for each question. Reliable Context's directly relevant observable guarantee is that any past record state can be reproduced with the sources that supported it at that time. The artifact should also state where its synthetic history or public benchmark differs from production records.

Conclusion

The decisive question for AI agent memory update is not whether the system remembers a related passage. It is whether the system can identify the claim that applies to the requested time, preserve what applied before, expose unresolved disagreement, decline unsupported questions, and reconstruct the evidence and state transition behind its answer.

A defensible test begins with two clocks, effective time and observation time. It then builds a minimal history containing an original claim, a successor, a conflict, a revocation, and an unobserved field. Present-state, past-state, change-since, knowledge-as-of, and unanswerable questions are asked over that same history. Results are reported as update adoption, stale and premature error rates, historical reconstruction, conflict disclosure, evidence reconstruction, and abstention precision and recall, each with its denominator.

Public benchmarks provide useful anchors but not interchangeable leaderboards. Dataset revision, history setting, reader, judge, prompt, source permissions, and question subset can materially change the result. A paired full-context baseline and an oracle-evidence diagnostic help localize failure, while a separate deletion suite avoids rewarding mutually incompatible behaviors. The production decision should remain conservative: if current value, temporal scope, transition, and exact evidence do not agree, the agent should abstain or expose the conflict rather than manufacture certainty.