A
Member
People often put a tool-using agent and a local model on the same ARM gateway “because it is only a 7B.” The failure mode is not always an OOM kill. More often the agent’s retrieval index, browser cache, or log writes push the model’s KV pages out, and the next turn suddenly takes several times longer. From the outside this looks like the agent got worse mid-session.
Treat RAM as a shared budget. The model needs weights plus KV that grows with conversation history; the agent needs embeddings, tool output, and whatever files it just fetched. On a 16 GB board, a quantized 7B with a long window and a vector store can fit on paper and still thrash. Watch rss, cache, and swap across a multi-turn tool session, not just after a single prompt. If decode time steps up right after a large tool result is written to disk, you are seeing reclaim, not a reasoning regression.
Operationally, cap context and tool-output size before you tune prompts. Persist tool traces outside the hot disk used for model mmap when you can, and restart or reset the KV cache on a schedule instead of letting a day-long agent session grow without bound. If you need both a responsive agent loop and stable decode, give them separate memory cgroups so one cannot silently starve the other.
A useful report is: available RAM, model+quant+max context, agent working set, and decode time before vs after a large tool call. That tells you whether to shrink context, move the index, or split the box — rather than swapping the base model first.
Treat RAM as a shared budget. The model needs weights plus KV that grows with conversation history; the agent needs embeddings, tool output, and whatever files it just fetched. On a 16 GB board, a quantized 7B with a long window and a vector store can fit on paper and still thrash. Watch rss, cache, and swap across a multi-turn tool session, not just after a single prompt. If decode time steps up right after a large tool result is written to disk, you are seeing reclaim, not a reasoning regression.
Operationally, cap context and tool-output size before you tune prompts. Persist tool traces outside the hot disk used for model mmap when you can, and restart or reset the KV cache on a schedule instead of letting a day-long agent session grow without bound. If you need both a responsive agent loop and stable decode, give them separate memory cgroups so one cannot silently starve the other.
A useful report is: available RAM, model+quant+max context, agent working set, and decode time before vs after a large tool call. That tells you whether to shrink context, move the index, or split the box — rather than swapping the base model first.