Silent-corruption testing for distributed collectives with rank-distinct data
Parent: Mac local LLMs: Clusters, RDMA, exo and ds4 · Published reference · snapshot 2026-10-05
↓ Facts as markdownall context files
Test dtypes matter: the 4443 report saw wrong gathers for int32 and float32; sums over bfloat16 hide small errors under rounding, so use int32 for exactness.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Facts
- Test dtypes matter: the 4443 report saw wrong gathers for int32 and float32; sums over bfloat16 hide small errors under rounding, so use int32 for exactness. [source]
- Ring all_reduce switches to one direction and one wire at 32768 bytes or less, mesh multi-wire starts at 512 KiB, and mesh switches algorithm at 32 KiB per wire when size is above 2, so a corruption test must cover sizes on both sides of each [source]
- Ring sum_scatter changes wire count at 65536 bytes [source]
- Ring all_gather splits every rank's region into `2 * n_wires` slices of `ceil(n_bytes / (2 * n_wires))` bytes, so tail clamping is exercised only when the size is not a multiple of `2 * n_wires` [source]
- With constant input, an all_gather or all_sum that moves the wrong slice returns the same values, so constant data cannot detect it [source]
- UC delivery never reports a lost message, so integrity tests need a timeout as well as an equality check [source]
- Use `MLX_JACCL_RING=1` (or `--backend jaccl-ring`) to test the ring path separately from the mesh path [source]
- Integer dtypes give exact expected sums, unlike bfloat16 [source]
Children
- No children recorded.