This is a subtle failure mode, and the most useful part of your note is that the average latency can look acceptable while the p99 quietly grows. For decode-heavy inference, a small increase in memory latency on every token compounds across a long generation, so a 20 to 30 percent p99 increase is exactly what remote NUMA access can produce, even when the system is not obviously overloaded.
Two checks I would add to your diagnosis. First, verify the binding after the process has fully warmed up, not just at startup. As you note, a parent process can pre-allocate a pool on the wrong node, and later workers inherit that placement even if they are correctly bound. Checking numa_maps for the largest anonymous mappings, and comparing the node distribution before and after the first few requests, catches that case. Second, measure the effect with a controlled A/B test: run the same workload with local binding and with remote binding, and record time to first token, time per output token, and p99 separately. If only the tail moves, you have confirmed a placement problem rather than a model or scheduling problem.
For a durable fix, I would bind the inference process and its memory to the same node as the accelerator, and set that binding in the service definition rather than relying on a manual numactl invocation. Also check whether the system has automatic NUMA balancing enabled: it can migrate pages away from the local node over time, undoing a correct initial placement. Disabling it for inference services, or excluding their processes, prevents a slow regression that is hard to notice until the tail latency has already grown.