JACCL RDMA completion status checking and silent corruption
Parent: Mac local LLMs: Clusters, RDMA, exo and ds4 · Published reference · snapshot 2026-10-05
↓ Facts as markdownall context files
zcbenz's closing reason: it is easier for maintainers to write and verify such changes themselves than to review contributor PRs over several rounds
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Facts
- zcbenz's closing reason: it is easier for maintainers to write and verify such changes themselves than to review contributor PRs over several rounds [source]
- angeloskath measured about 10% slowdown (about 1 us) on 16 KB 4-way all-reduce and held the PR for a JACCL refactor and optimisation pass, noting the team treats Thunderbolt as a reliable channel [source]
- The rebased PR changed 3 files (+61 lines) and added 4 check sites in mesh_impl.h and 5 in ring_impl.h [source]
- The first commit message cites IBV_WC_RETRY_EXC_ERR as an example of an RDMA transport error that was silently ignored [source]
- The original PR also made Connection::poll() and the free poll() throw on a negative ibv_poll_cq return [source]
- The PR author tested 256, 512, 5478+433 and 8490+512 token generations on the 612 GB model with zero RDMA errors after the fix, and saw no regression on models under 10 GB per rank [source]
- The PR author states small models are unaffected because their RDMA operations always succeed, so the bug is latent [source]
- After a peer is SIGKILLed, ibv_poll_cq returned 0 about 2.64 billion times, never negative, never a bad wc.status, so status checking cannot detect peer loss [source]
- Related exo report: macOS 26.3.1 4x M3 Ultra clusters hit "[jaccl] Recv failed with errno=2", errno=60 and "Changing queue pair to RTR failed with errno 22"; the reporter says the #3451 race fix removed the post-kill RTR failures [source]
- Issue #3467 root cause: after the #3412 refactor Connection::info() only accepts IPv4-mapped GIDs (::ffff:x.x.x.x) but Apple Thunderbolt RDMA can expose only link-local fe80:: GIDs, leaving the GID uninitialised and giving RTR errno 22 [source]
Corrections and disagreements
- CONTRADICTS distributed-inference-across-macs.md: PR #3152 is closed, not open; zcbenz closed it without merging on 2026-10-02 [source]
Children
- No children recorded.