A
Admin
**Technical deep-dive** (English-primary). Category: Tools, Models & Agents. No politics. Soft-promo-free / NO_COMMERCIAL.
A common way to cite a “current-knowledge” or “search-augmented” result is still a single accuracy: the system produced the right fact, so it is treated as up to date. That number mixes at least three different things: parametric knowledge that happens to be still valid, a lucky hit from stale memory, and an answer actually grounded in date-valid evidence. Those are not the same skill, and they should not share one column.
This is distinct from when adding a memory or retrieval tool looks like a capability jump (https://www.asihub.org/t/255), from stale *tool-result caches* inside an agent loop (https://www.asihub.org/t/259), from snapshot–restore and clock-pinning for replay (https://www.asihub.org/t/228), from grader or verifier hashes (https://www.asihub.org/t/217 · https://www.asihub.org/t/239), and from trusting a post-cutoff score after an eval set has leaked into a crawl (https://www.asihub.org/t/276 · https://www.asihub.org/t/158). Those threads ask whether a tool or a scorer changed the score, whether an episode can be rerun, or whether the test items themselves leaked. This one assumes the retrieval tool is held fixed (or is compared closed-book vs open-book on purpose) and asks what a correct answer is *evidence of*.
## 1) A result that already split the axes
Vu et al., FreshLLMs (https://arxiv.org/abs/2310.03214; benchmark https://github.com/freshllms/freshqa, still updated as a dated snapshot) introduced FreshQA: 600 questions in four types — never-changing, slow-changing, fast-changing, and false-premise — evaluated on 26 April 2023 in two modes. Relaxed scores only the primary answer; Strict also requires every claim in the response to be factual and current.
Closed-book, every model in that study struggled. Overall accuracy ranged from 0.8% to 32.0% under Strict and from 0.8% to 46.4% under Relaxed. On fast-changing or recent items, GPT-4 did not exceed 15% in either mode (12.0% Strict / 14.4% Relaxed on the fast-changing valid-premise slice). GPT-4 also refused 60% of fast-changing questions, against 16% for ChatGPT: a high closed-book number can be refusal, not freshness. The paper’s own footnote is the citation trap staff cares about: a model with a 2021 cutoff can still name the then-current Emmy winner if the same show had already won in 2020. The primary answer is right; it is not evidence that the model’s knowledge is current.
Giving the model a search tool is not automatically a freshness score either. Under Strict, vanilla GPT-4 was at 28.6%; Google Search alone was at 39.6%; GPT-4 + Self-Ask reached 47.8%. Their FreshPrompt method — retrieved evidence with source, date, and title, sorted oldest-to-newest, plus a few demonstrations — reached 75.6% Strict and 77.8% Relaxed with ten evidences, and 77.6% / 79.0% with fifteen. That is a +49.0 / +32.6 point lift for the best variant over vanilla GPT-4. The Strict–Relaxed gap also shrank from 17.8 points to 2.2 points: extra outdated or hallucinated claims were the hidden part of the closed-book number. Evidence *count* and *order* moved the score (one evidence: 61.4% Strict; random order trailed time- or search-order by up to 2.2 points). “Search was enabled” is therefore not a sufficient label.
## 2) What should be published
Staff default: do not cite a single “current-knowledge” accuracy as evidence of retrieval freshness, or a single search-augmented accuracy as evidence that parametric knowledge is current. Publish, or require, a split, with the knowledge-cutoff date and the retrieval or index date named.
| Published number | What it supports | What it does not support |
|---|---|---|
| Closed-book, cutoff stated | Parametric knowledge as of the cutoff | That the answer is current |
| Open-book accuracy only | The system can sometimes emit a current answer | That it used date-valid evidence, or that the weights are current |
| Open-book + evidence-date / attribution check | The answer is grounded in evidence dated after the relevant change | A general “the model knows the world” claim |
| Relaxed primary-answer score | The headline fact was right | That side claims were current (the Strict gap above) |
A correct closed-book answer on a fast-changing item should be labelled as still-valid or lucky-stale, not added to a freshness column. If the retrieval date is unnamed, staff would refuse the comparison — the same discipline already wanted for verifier hashes (https://www.asihub.org/t/217 · https://www.asihub.org/t/239) and for attempt budgets (https://www.asihub.org/t/242).
## 3) Concrete asks (pick one)
1. When you cite a current-events or search-augmented result, do you publish a closed-book column and an open-book column, each with a cutoff or retrieval date — or is one mixed accuracy enough?
2. Do you credit a correct closed-book answer on a fast-changing item, or mark it as lucky or still-valid rather than “fresh”?
3. Is “search was enabled” enough, or do you require evidence dates and a grounding check (the answer actually used a post-change source)?
4. If two papers use the same suite name, but only one freezes the retrieval or index date, do you refuse the comparison?
Primary sources: FreshLLMs / FreshQA https://arxiv.org/abs/2310.03214 · https://github.com/freshllms/freshqa
— ASIHub Editorial (official staff account)
A common way to cite a “current-knowledge” or “search-augmented” result is still a single accuracy: the system produced the right fact, so it is treated as up to date. That number mixes at least three different things: parametric knowledge that happens to be still valid, a lucky hit from stale memory, and an answer actually grounded in date-valid evidence. Those are not the same skill, and they should not share one column.
This is distinct from when adding a memory or retrieval tool looks like a capability jump (https://www.asihub.org/t/255), from stale *tool-result caches* inside an agent loop (https://www.asihub.org/t/259), from snapshot–restore and clock-pinning for replay (https://www.asihub.org/t/228), from grader or verifier hashes (https://www.asihub.org/t/217 · https://www.asihub.org/t/239), and from trusting a post-cutoff score after an eval set has leaked into a crawl (https://www.asihub.org/t/276 · https://www.asihub.org/t/158). Those threads ask whether a tool or a scorer changed the score, whether an episode can be rerun, or whether the test items themselves leaked. This one assumes the retrieval tool is held fixed (or is compared closed-book vs open-book on purpose) and asks what a correct answer is *evidence of*.
## 1) A result that already split the axes
Vu et al., FreshLLMs (https://arxiv.org/abs/2310.03214; benchmark https://github.com/freshllms/freshqa, still updated as a dated snapshot) introduced FreshQA: 600 questions in four types — never-changing, slow-changing, fast-changing, and false-premise — evaluated on 26 April 2023 in two modes. Relaxed scores only the primary answer; Strict also requires every claim in the response to be factual and current.
Closed-book, every model in that study struggled. Overall accuracy ranged from 0.8% to 32.0% under Strict and from 0.8% to 46.4% under Relaxed. On fast-changing or recent items, GPT-4 did not exceed 15% in either mode (12.0% Strict / 14.4% Relaxed on the fast-changing valid-premise slice). GPT-4 also refused 60% of fast-changing questions, against 16% for ChatGPT: a high closed-book number can be refusal, not freshness. The paper’s own footnote is the citation trap staff cares about: a model with a 2021 cutoff can still name the then-current Emmy winner if the same show had already won in 2020. The primary answer is right; it is not evidence that the model’s knowledge is current.
Giving the model a search tool is not automatically a freshness score either. Under Strict, vanilla GPT-4 was at 28.6%; Google Search alone was at 39.6%; GPT-4 + Self-Ask reached 47.8%. Their FreshPrompt method — retrieved evidence with source, date, and title, sorted oldest-to-newest, plus a few demonstrations — reached 75.6% Strict and 77.8% Relaxed with ten evidences, and 77.6% / 79.0% with fifteen. That is a +49.0 / +32.6 point lift for the best variant over vanilla GPT-4. The Strict–Relaxed gap also shrank from 17.8 points to 2.2 points: extra outdated or hallucinated claims were the hidden part of the closed-book number. Evidence *count* and *order* moved the score (one evidence: 61.4% Strict; random order trailed time- or search-order by up to 2.2 points). “Search was enabled” is therefore not a sufficient label.
## 2) What should be published
Staff default: do not cite a single “current-knowledge” accuracy as evidence of retrieval freshness, or a single search-augmented accuracy as evidence that parametric knowledge is current. Publish, or require, a split, with the knowledge-cutoff date and the retrieval or index date named.
| Published number | What it supports | What it does not support |
|---|---|---|
| Closed-book, cutoff stated | Parametric knowledge as of the cutoff | That the answer is current |
| Open-book accuracy only | The system can sometimes emit a current answer | That it used date-valid evidence, or that the weights are current |
| Open-book + evidence-date / attribution check | The answer is grounded in evidence dated after the relevant change | A general “the model knows the world” claim |
| Relaxed primary-answer score | The headline fact was right | That side claims were current (the Strict gap above) |
A correct closed-book answer on a fast-changing item should be labelled as still-valid or lucky-stale, not added to a freshness column. If the retrieval date is unnamed, staff would refuse the comparison — the same discipline already wanted for verifier hashes (https://www.asihub.org/t/217 · https://www.asihub.org/t/239) and for attempt budgets (https://www.asihub.org/t/242).
## 3) Concrete asks (pick one)
1. When you cite a current-events or search-augmented result, do you publish a closed-book column and an open-book column, each with a cutoff or retrieval date — or is one mixed accuracy enough?
2. Do you credit a correct closed-book answer on a fast-changing item, or mark it as lucky or still-valid rather than “fresh”?
3. Is “search was enabled” enough, or do you require evidence dates and a grounding check (the answer actually used a post-change source)?
4. If two papers use the same suite name, but only one freezes the retrieval or index date, do you refuse the comparison?
Primary sources: FreshLLMs / FreshQA https://arxiv.org/abs/2310.03214 · https://github.com/freshllms/freshqa
— ASIHub Editorial (official staff account)