ibv_devinfo hang as an RDMA health probe
Parent: Mac local LLMs: Clusters, RDMA, exo and ds4 · Published reference · snapshot 2026-10-05
↓ Facts as markdownall context files
The probe idea: run `ibv_devinfo` and treat a hang as a sign that Thunderbolt RDMA is wedged, then show a dashboard warning. It comes from one sentence in exo issue 1712.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Facts
- The probe idea: run `ibv_devinfo` and treat a hang as a sign that Thunderbolt RDMA is wedged, then show a dashboard warning. It comes from one sentence in exo issue 1712. [source]
- The issue says jaccl errors such as exo issue 1711 recover only by unplugging and replugging the Thunderbolt 5 cable or rebooting, and that "usually when these errors happen, the `ibv_devinfo` command hangs". It asks for a dashboard warning with advice to reconnect cables, then reboot. [source]
- The issue has one participant, no comments, no assignee, no linked branch or pull request, and the label `enhancement`. It gives no logs, no macOS version and no reproduction. [source]
- ds4's docs use `ibv_devinfo -v` the other way round: as a static check that the port is active and has an IPv4-mapped GID, and as the place to read `--rdma-device` and `--rdma-gid-index` values. They do not mention hangs. [source]
- Exo's reliability PRs 2367 and 2370 took other routes to the same goal. PR 2367 stops a runner when its supervisor sees MLX's ring diagnostic "Too many send/recv errors. Aborting..." on stderr. PR 2370 stops the first-rank runner after 90 seconds without a token or prefill progress report while a generation is in flight. Neither uses `ibv_devinfo`. [source]
- Both PRs target a stuck runner process. A wedged driver that hangs `ibv_devinfo` is a different layer, and a runner blocked inside JACCL would already be caught only by the 90-second rule. [source]
- A hung probe may not be killable: processes stuck in the Thunderbolt RDMA stack have been seen in uninterruptible sleep (`Us+`) where `kill -9` does nothing. A probe run without a timeout, or with one that only sends a signal, can leave stuck probe processes behind. [source]
- `ibv_devinfo` reports `PORT_ACTIVE` only when all connected Macs have RDMA enabled (per distributed-inference-across-macs.md), so a clean exit with `PORT_DOWN` and a hang are different signals. [source]
- Probe versus progress watchdog. Issue 1712 proposes a link-layer probe; the merged-or-open reliability PRs use runner-level signals (stderr diagnostic, token silence) and recreate the runner. The sources do not compare them. [source]
- Whether `ibv_devinfo` hangs on macOS 26.5 and later, or on macOS 27's reworked library. No fetched source tests it after March 2026. [source]
- A safe probe form: a short-deadline child process that is expected to be abandoned if it hangs, plus a count of stuck probes. [source]
- exo issue 1712 is still open with no assignee, comment, branch or pull request. [source]
- Exo issue 1712's only evidence is one sentence from the author that `ibv_devinfo` usually hangs when jaccl errors occur; it has no log or repro. [source]
- The issue asks for a dashboard warning that suggests reconnecting cables, then rebooting. [source]
- ds4 documents `ibv_devinfo -v` as a static link check, not as a hang probe. [source]
- exo PR 2367 detects a failed MLX ring by the stderr text "Too many send/recv errors. Aborting..." and stops that runner. [source]
- exo PR 2370 stops a first-rank runner after 90 seconds of silence during a generation request, and checks every 5 seconds. [source]
- Neither PR 2367 nor PR 2370 uses `ibv_devinfo`. [source]
- Processes stuck in the Thunderbolt RDMA stack have been reported in uninterruptible sleep where `kill -9` has no effect, so a hanging probe needs an abandon-on-timeout design. [source]
- `ibv_devinfo` shows `PORT_ACTIVE` only when every connected Mac has RDMA enabled, per the existing dossier distributed-inference-across-macs.md. [source]
Children
- No children recorded.