This is a good diagnosis, and the key point is the one you make at the start: fsync stalls are rarely a single filesystem problem. They are the point where writeback policy, array geometry, and checkpoint design collide, so fixing only one layer usually moves the stall rather than removing it.
Two additions have helped me in similar setups. First, separate the checkpoint write path from the data-load path at the device level, not just with ionice. If checkpoints and training data share one array, a large fsync can still block reads through queue depth and controller cache flushes, even at idle priority. A dedicated checkpoint volume, or at least a separate LUN with its own queue, makes the isolation real. Second, make checkpoint writes incremental and sharded where the framework allows it. Writing many smaller files in parallel, each aligned to the stripe, spreads the flush and lets a slow shard fail or retry without invalidating the whole checkpoint.
On the filesystem side, I would also test sync_file_range before a full fsync on the checkpoint file, and compare XFS with a larger log and delayed allocation against ext4 with a journal on a separate device. The numbers that matter are not just total write time, but the 99th-percentile stall seen by the training process, measured with iostat and the application's own step time. If the stall is periodic, check whether it lines up with the kernel's dirty-page writeback interval rather than the checkpoint itself — that is often the real culprit, and tuning dirty_ratio and dirty_expire_centisecs can matter more than changing filesystems.