Larger retrieval windows can reduce fact accuracy — expect more triage time

Increasing the retrieved-evidence budget can raise verification and bug-fixing work and increase factual risk: the paper shows accuracy falls when retrieval grows and that stale, competing statements cause most failures.

What actually changed

A fresh arXiv submission evaluates MemoryAgentBench’s single-hop FactConsolidation task and finds that increasing the retrieved evidence budget hurts some models. With Qwen2.5-7B-Instruct and a 6K-history BM25 index, accuracy fell from 45% to 32% when the retrieved-evidence budget rose from 64 tokens to 512 tokens. A 32K-history replication again shows BM25-64 outperforming BM25-512 (69% versus 58%). The authors audited retrieved sets and found the newest statement present in each set, yet errors persisted — the paper summarises this as "retrieving the correct fact is not enough." Most misanswers occur when a question also contains an obsolete, competing statement; in a controlled swap where one unrelated statement was replaced by a stale competitor, Qwen2.5’s accuracy dropped from 100% to 44.6%, and Qwen3-14B’s from 100% to 87.8%.

Who it affects

• Teams using retrieval-augmented agents or memory-agent setups, especially with BM25-style retrieval.
• Projects that scale retrieval windows or increase the token budget under the assumption that more evidence increases correctness.
• Deployments that maintain long histories (the paper reports results on 6K and 32K history settings) and where facts can be superseded over time.

The effects are demonstrated on Qwen2.5-7B-Instruct and Qwen3-14B in the authors’ experiments; the paper does not claim broader model coverage beyond those runs.

What it costs or what it replaces

• Pricing: the announcement does not state pricing. The paper is a research result, not a commercial product, so there is no stated cost to acquire or run a tool here.

• What it replaces: the findings caution against replacing smaller, precision-focused retrieval budgets with larger evidence windows without further safeguards. In the experiments, BM25-64 outperformed BM25-512 in multiple settings, so simply increasing retrieved tokens is not a guaranteed improvement.

• Operational cost: the numerical results imply a hidden cost — larger retrieved windows can introduce stale, competing statements that materially lower answer accuracy. That will translate into more time spent diagnosing failures, adding validation checks, and designing mechanisms to detect or resolve obsolescence if you adopt large-budget retrieval.

What we don't know

• Whether these failure modes generalise to retrieval strategies other than BM25 (the paper tests BM25 settings).
• How models beyond Qwen2.5-7B-Instruct and Qwen3-14B behave under the same conditions.
• Which mitigation techniques restore accuracy (for example, ranking tweaks, recency heuristics, or explicit stale-detection) and how costly they are to implement.
• Real-world impact across diverse tasks beyond the single-hop FactConsolidation benchmark.

What to do next

1. Run a controlled A/B test replicating the authors’ variable: compare your system’s performance with a small retrieved-evidence budget (e.g., 64 tokens) versus a much larger one (e.g., 512 tokens) on a representative set of historical questions.
2. Perform a gold-assisted post-hoc audit like the paper’s: confirm whether the newest statement appears in retrieved sets, and track how often stale competing statements coincide with errors.
3. If larger budgets hurt accuracy, keep retrieval budgets conservative and prioritise fixes that detect or demote stale/competing statements (for example, recency filters or explicit version checks) before scaling evidence size.