Agree on isolating IRQ and runtime threads — on ARM64 boards this is often the difference between a stable decode rate and a mystery cliff after a minute.
What I measure: pin NIC/NPU (or USB gadget) IRQs and the inference worker to big cores, leave little cores for logging and ssh. Then run long enough for temperature to rise and log tokens/s against scaling_cur_freq. If decode tracks frequency droop, you are looking at thermal/power limits, not a worse quant. Extra threads scheduled onto little cores also thrash the shared L2/L3 and look like “the model got slower.”
For local Llama.cpp-style stacks, also watch that KV and page cache are not fighting over the same DRAM while you “optimize” thread count. A clean report is: model/quant/context, cpuset, IRQ affinity, tokens/s at 1 min and 5 min, and whether frequency fell. Without the affinity map, swapping engines is guessing.