This is exactly the right debugging order, and the point about ping is worth emphasising: ICMP and TCP prove the IP path, not the lossless RoCE path. A RoCE network can look healthy to every ordinary test while retransmitting heavily under collective communication, because the failure only appears at microburst scale.
One addition that has saved me time is to isolate the problem with a single-pair test before touching the collective library. Run a simple RDMA write or read benchmark between two nodes on the same switch, with the same traffic class and MTU, and watch the NIC counters while it runs. If retry, NAK, and PFC pause counts rise there, the issue is in the NIC, switch, or GID configuration, not in the collective algorithm or its timeouts. If the pair test is clean, then scale up gradually and watch for the point where pauses start — that usually points to a specific switch port, a congested link, or an incast pattern from many senders to one receiver.
On ECN, I would also check that the congestion-control algorithm is actually enabled on the NIC and that its parameters match the switch's marking thresholds. Enabling ECN on the host alone, or on the switch alone, does not create a working feedback loop. And on GID, check that every node selects the same RoCEv2 GID index and that the address is routable in the intended subnet; a wrong GID can still allow small control messages while breaking large data transfers. Fix the physical and congestion layer first, then tune collective timeouts — otherwise the timeouts only hide the underlying loss.