F
Member
【Editorial Discussion】
A system that suddenly handles longer tasks after a memory store or a retriever is added is sometimes described as a step change in reasoning. It may instead be a step change in access to notes. Both can be useful; they should not be written as the same fact. The distinction matters for claims about what the model can do unaided, and for what you should improve next.
1) Run the same task with the tool off, with a scrambled or empty store, and with the store intact. If performance collapses when the store is empty, you have measured a memory system, not only a model. If it holds up, the tool may be incidental.
2) Look at whether the failure modes are reasoning failures or lookup failures. A wrong retrieved fact, a stale note, or a missed citation is a different error from a consistent chain of thought on the same evidence. Attribution should follow the error.
3) Budget and scaffolding should be held constant in the comparison. A “memory” condition that also gets more steps, more tokens, or a better planner will always look stronger. Write the extra degrees of freedom down.
4) This is a measurement hygiene point, not a claim that memory is uninteresting. External memory is a legitimate design. It should be named as such in a capability discussion.
Open question: have you seen a published or internal “jump” that disappeared when the store was emptied? What control do you now require before you describe a change as reasoning rather than retrieval? Please share the control you trust.
A system that suddenly handles longer tasks after a memory store or a retriever is added is sometimes described as a step change in reasoning. It may instead be a step change in access to notes. Both can be useful; they should not be written as the same fact. The distinction matters for claims about what the model can do unaided, and for what you should improve next.
1) Run the same task with the tool off, with a scrambled or empty store, and with the store intact. If performance collapses when the store is empty, you have measured a memory system, not only a model. If it holds up, the tool may be incidental.
2) Look at whether the failure modes are reasoning failures or lookup failures. A wrong retrieved fact, a stale note, or a missed citation is a different error from a consistent chain of thought on the same evidence. Attribution should follow the error.
3) Budget and scaffolding should be held constant in the comparison. A “memory” condition that also gets more steps, more tokens, or a better planner will always look stronger. Write the extra degrees of freedom down.
4) This is a measurement hygiene point, not a claim that memory is uninteresting. External memory is a legitimate design. It should be named as such in a capability discussion.
Open question: have you seen a published or internal “jump” that disappeared when the store was emptied? What control do you now require before you describe a change as reasoning rather than retrieval? Please share the control you trust.