A
Member
On an ARM64 dev board or mini PC, a local Llama.cpp-style runtime often looks fine for the first 30–60 seconds and then drops. The usual next step is a smaller model or a more aggressive quant. Before that, check whether the SoC has simply left its boost frequency.
Measure prefill and decode separately, and keep the run long enough for temperature to rise. Log tokens/s, CPU/NPU frequency, and board temperature on the same timeline. If decode speed tracks frequency droop, you are looking at thermal or power-limit throttling, not at a worse checkpoint. Big.LITTLE layouts make this easier to miss: extra threads scheduled onto little cores can lower the average while also increasing cache traffic.
Memory is the other silent limiter. These boards usually share DRAM among weights, the KV cache, and the OS page cache. A longer context grows KV and starts reclaiming cache or swapping; that shows up as a sudden decode stall rather than a clean OOM. Compare resident memory and swap against context length before blaming the quant format. Pinning inference threads to the big cores, and not compiling or browsing on the same machine during the benchmark, is often a larger win than dropping another bit of precision.
Publish comparisons with model, quant, context, thread pin, tokens/s at 1 minute and at 5 minutes, temperature, and whether frequency fell. A single peak tokens/s number on a cold board is not a useful ARM result.
Measure prefill and decode separately, and keep the run long enough for temperature to rise. Log tokens/s, CPU/NPU frequency, and board temperature on the same timeline. If decode speed tracks frequency droop, you are looking at thermal or power-limit throttling, not at a worse checkpoint. Big.LITTLE layouts make this easier to miss: extra threads scheduled onto little cores can lower the average while also increasing cache traffic.
Memory is the other silent limiter. These boards usually share DRAM among weights, the KV cache, and the OS page cache. A longer context grows KV and starts reclaiming cache or swapping; that shows up as a sudden decode stall rather than a clean OOM. Compare resident memory and swap against context length before blaming the quant format. Pinning inference threads to the big cores, and not compiling or browsing on the same machine during the benchmark, is often a larger win than dropping another bit of precision.
Publish comparisons with model, quant, context, thread pin, tokens/s at 1 minute and at 5 minutes, temperature, and whether frequency fell. A single peak tokens/s number on a cold board is not a useful ARM result.