Health monitoring, alerting and safe automated recovery for a Thunderbolt eGPU

Parent: Thunderbolt eGPU on Linux for local LLM inference · Published reference · snapshot 2026-09-08 · skill devops-linux-internals/references/egpu-health-monitoring-and-automated-recovery-linux.md

Also known as: egpu auto recovery, egpu monitoring, egpu watchdog, egpu-reinit automation, liveness probe gpu, node_exporter textfile, nvidia xid alert, systemd OnFailure

↓ Facts as markdown all context files

Watching a Thunderbolt eGPU on a headless Linux box and recovering it safely — the kernel-log and sysfs signatures worth alerting on, a guarded liveness probe that tells a dead GPU from a bridge with

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Health monitoring, alerting and safe automated recovery for a Thunderbolt eGPU

Watching a Thunderbolt eGPU on a headless Linux box and recovering it safely - the kernel-log and sysfs signatures worth alerting on, a guarded liveness probe that tells a dead GPU from a bridge with memory decoding off, alert-only systemd templates, and a gated recovery state machine with rate limits, lockout and a break-glass path, defaulting to alert-only. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#health-monitoring-alerting-and-safe-automated-recovery-for-a-thunderbolt-egpu]
--- name: egpu-health-monitoring-and-automated-recovery-linux title: Thunderbolt eGPU health monitoring, alerting and safe automated recovery (headless Linux LLM box) description: TRIGGER when designing or debugging monitoring/alerting for a Thunderbolt/USB4 NVIDIA eGPU on a headless Linux host, choosing kernel-log/sysfs/NVML signals, writing an alert-only systemd watchdog, a config-space+BAR0 liveness probe, node_exporter textfile metrics, or a rate-limited auto re-init with lockout and break-glass. SKIP root-cause triage (linux-nvidia-egpu-fallen-off-bus-diagnosis.md), loader/boot ordering and manual re-init (linux-egpu-hotplug-boot-orchestration.md), safe detach (egpu-hot-unplug-pciehp-safety-linux.md), AER/DPC tuning (pcie-power-management-aer-dpc-egpu-linux.md), PSU/thermals (egpu-power-enclosure-and-thermals-linux.md). --- [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#health-monitoring-alerting-and-safe-automated-recovery-for-a-thunderbolt-egpu]

Thunderbolt eGPU health monitoring and automated recovery (Linux)

Verified-as-of 2026-09-25. Context: RTX 5080 in a Razer Core X V2 over TB4 on an Intel NUC 15 Pro, Ubuntu 26.04.1, kernel 7.0.0-34, nvidia open driver 610.57.04, loaded late by egpu-nvidia.service. Tags: [SOURCED url] fetched and read this session; [INFERRED] reasoning or general Linux knowledge; [UNVERIFIED] could not be confirmed here. Nothing in this file was tested on the box. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#thunderbolt-egpu-health-monitoring-and-automated-recovery-linux]
Sibling references (not re-covered): linux-nvidia-egpu-fallen-off-bus-diagnosis.md (signature triage), linux-egpu-hotplug-boot-orchestration.md (loader, ordering, re-init, safe detach), egpu-hot-unplug-pciehp-safety-linux.md (removal safety), pcie-power-management-aer-dpc-egpu-linux.md (AER/DPC reading), egpu-power-enclosure-and-thermals-linux.md (power vs bus faults, soak tests). [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#thunderbolt-egpu-health-monitoring-and-automated-recovery-linux]
Terms: recovery (also re-init) is the mutating remove/rescan/reload sequence; the probe only reads; the gate decides whether recovery may start. Commands that touch /sys, PCI config space, /var/lib or systemd need root, and journalctl -k needs root or journal-group membership (systemd-journal or adm) [UNVERIFIED]; examples use sudo. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#thunderbolt-egpu-health-monitoring-and-automated-recovery-linux]

Core Concepts

Detect cheaply, act rarely. Monitoring (read-only, every minute) and recovery (mutating, rare, gated) are separate units with separate privileges. Default is alert-only. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#core-concepts]
Three independent evidence planes must be combined before any action: kernel log (events), sysfs (state), NVML/nvidia-smi (driver-level view). A single plane lies: nvidia-smi hangs when the GPU is wedged; sysfs can look fine while BAR0 reads 0xffffffff; the log is edge-triggered and can rotate away. The templates below use two planes (probe state, and a kernel-log Link Down check in the gate); the NVML plane is described but not implemented. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#core-concepts]
Distinguish "GPU dead" from "bridge Mem-". On this box BAR0 once read 0xffffffff while config space answered because a Thunderbolt host reset left bridge Memory decode off. A deterministic probe (vendor ID, bridge COMMAND and, when enabled, BAR0 chip ID) separates a 1-second fix (set COMMAND Mem+BusMaster) from a real dead card. [INFERRED from box history] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#core-concepts]
Xid 154 is a summary, not a cause. It reports the recovery action required for other Xids; the recovery action for Xid 79 is a bare-metal restart, for 119/120 a GPU reset plus software investigation. [SOURCED https://docs.nvidia.com/deploy/xid-errors/analyzing-xid-catalog.html] So a software "re-init" is a best-effort attempt, and NVIDIA's own guidance for 79 is reboot-class. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#core-concepts]
Automation must never do a surprise removal. A card that is physically gone (enclosure off, cable pulled, pciehp Link Down with no re-presence) must be treated as "absent", not "broken": alert, do not rescan/remove loops. See egpu-hot-unplug-pciehp-safety-linux.md. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#core-concepts]
Bounded, observable, reversible. Rate limit, exponential backoff, lockout after N failures, a state file, an off switch, and a break-glass manual path are part of the design, not add-ons. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#core-concepts]
Alert on absence of data. A watchdog that stops running looks identical to a healthy box. Use a heartbeat metric with a timestamp and alert on staleness. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#core-concepts]

Signals to Watch

Signal to meaning table. Meanings are hypotheses to feed triage, not verdicts. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#signals-to-watch]

Log Signatures

Kernel log access. journalctl -k shows kernel messages; -f follows; -p filters priority. Persist logs across reboots with Storage=persistent in journald.conf (or create /var/log/journal), otherwise the very reboot that a failure causes erases the evidence. [INFERRED from journald.conf(5); man page fetch failed this session, so directive spelling is [UNVERIFIED] here but long-standing] Confirm with man journald.conf, then check that journalctl --list-boots shows earlier boots after a reboot [UNVERIFIED]. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#log-signatures]
Useful, cheap filters (read-only): [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#log-signatures]
-g/--grep uses PCRE2 and needs journalctl built with it [INFERRED; verify with journalctl --help]. Confirm with journalctl --help | grep -e --grep [UNVERIFIED]. A -g failure and a no-match can both exit non-zero, so nothing here reads its exit status as a verdict (the gate pipes journal output into grep). [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#log-signatures]
Kernel AER log shape. Example kernel output: PCIe Bus Error: severity=Uncorrectable (Fatal), type=Transaction Layer followed by device [vendor:device] error status/mask=... and a decoded bit line [SOURCED https://docs.kernel.org/PCI/pcieaer-howto.html]. Match on AER: / PCIe Bus Error and severity words, not exact spacing (format drifts across kernels) [INFERRED]. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#log-signatures]
Event-driven vs polling. For an event-triggered alert, a small unit that runs journalctl -k -f and greps is simplest; the alternative (systemd path unit on a state file, or a timer that scans --since) survives restarts better. The templates below poll sysfs and config space on a timer and do not scan the log for Xid or AER lines, so the log-only signals above need a matcher you add; the only log reads here are the gate's Link Down check and the alert excerpt. logcheck/rsyslog matchers work too, but journald is already the source here; add them only if mail-from-logcheck is already deployed [INFERRED]. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#log-signatures]
OnFailure= behaviour. OnFailure= activates listed units when the unit enters the failed state; for Type=oneshot a non-zero exit is failure unless SuccessExitStatus= says otherwise [SOURCED https://man7.org/linux/man-pages/man5/systemd.unit.5.html]. Design consequence: make the probe exit non-zero on "unhealthy" and let OnFailure= fire the notifier. StartLimitIntervalSec=/StartLimitBurst= cap how often a unit may be started; excess starts are refused [SOURCED same]; note that a timer-triggered oneshot that keeps failing will hit the start limit and stop being started, which for an alert-only probe is a silent-monitoring hazard, so set StartLimitIntervalSec=0 on the probe (heartbeat metric covers absence) [INFERRED]. At the 60 s cadence below, though, the default limit (from memory, 5 starts in 10 s) would not trip, so the directive is defensive [UNVERIFIED]. Each failing run re-enters the failed state and, as the man page reads, triggers OnFailure= again, so the alert unit rate-limits itself [UNVERIFIED]. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#log-signatures]

sysfs and NVML Health Probes

sysfs (per PCI function under /sys/bus/pci/devices/<BDF>/). [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sysfs-and-nvml-health-probes]
- power_state (D0, D1, D2, D3hot, D3cold, unknown, error) and d3cold_allowed [SOURCED https://raw.githubusercontent.com/torvalds/linux/master/Documentation/ABI/testing/sysfs-bus-pci]. Reading power_state of a D3cold device does not wake it [INFERRED; do not rely on this, read the bridge-side state instead]. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sysfs-and-nvml-health-probes]
- current_link_speed, current_link_width (and max_link_*) exist for PCIe devices [INFERRED from long-standing sysfs; not shown in the ABI page fetched here, so [UNVERIFIED] against 7.0]. Confirm with cat /sys/bus/pci/devices/<BDF>/current_link_speed on the box. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sysfs-and-nvml-health-probes]
- power/runtime_status [SOURCED sysfs-devices-power above]. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sysfs-and-nvml-health-probes]
- AER counters: the kernel AER document states counters are exposed as sysfs attributes and points at ABI file testing/sysfs-bus-pci-devices-aer_stats [SOURCED https://docs.kernel.org/PCI/pcieaer-howto.html]. The attribute names aer_dev_correctable, aer_dev_nonfatal, aer_dev_fatal (per device) and aer_rootport_total_err_ (root ports) are [INFERRED from the task brief and memory; the ABI file returned 404 at the path tried, so exact names and per-line format are [UNVERIFIED]]. Confirm with ls /sys/bus/pci/devices/<BDF>/ | grep aer and cat one file on the box, snapshot it, then export the TOTAL_ERR_ style lines as counters. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sysfs-and-nvml-health-probes]
- Also worth reading: bridge power/runtime_status (a suspended bridge makes downstream reads misleading) [INFERRED]. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sysfs-and-nvml-health-probes]
NVML/nvidia-smi. All calls wrapped in timeout 10. Cheap fields: nvidia-smi --query-gpu=name,pci.bus_id,temperature.gpu,power.draw,clocks.sm,pcie.link.gen.current,pcie.link.width.current,utilization.gpu,memory.used --format=csv,noheader (field names are stable in nvidia-smi --help-query-gpu [INFERRED; verify on box]). nvidia-smi dmon streams per-second samples for soak tests. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sysfs-and-nvml-health-probes]
- nvidia_gpu_exporter (community) parses nvidia-smi output (or optionally a native NVML backend), listens on port 9835, metrics prefixed nvidia_smi_ (example nvidia_smi_temperature_gpu), and lists GeForce/RTX among supported hardware [SOURCED https://github.com/utkuozdemir/nvidia_gpu_exporter]. The README states the native backend adds Xid error counters; verify that claim on this driver before relying on it [UNVERIFIED for 610.x + GeForce]. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sysfs-and-nvml-health-probes]
- dcgm-exporter exposes DCGM fields (e.g. DCGM_FI_DEV_GPU_TEMP) on port 9400 with CSV-configured counters [SOURCED https://github.com/NVIDIA/dcgm-exporter]. Which fields DCGM supports on a GeForce board (as opposed to data-center boards) was not established: [UNVERIFIED]. Prefer the nvidia-smi based exporter for GeForce unless DCGM is confirmed to run. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sysfs-and-nvml-health-probes]
- Caution: any exporter that polls NVML keeps the GPU awake and can hang when the GPU is wedged; a hung scrape is itself a health signal. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sysfs-and-nvml-health-probes]

Liveness Probe

Goal: cheap, deterministic, no writes, no driver dependence. Order matters, each step guards the next. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#liveness-probe]
- Presence: [ -e /sys/bus/pci/devices/$GPU ]. If missing: state ABSENT -> alert, stop. Do not rescan (see egpu-hot-unplug-pciehp-safety-linux.md). [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#liveness-probe]
- Config vendor ID: setpci -s $GPU 0.w should equal 10de (NVIDIA). ffff -> UNREACHABLE (link dead or removed): alert, stop. Any other or missing answer (no setpci, permission error) -> UNKNOWN: alert, stop. Config reads use a different path than memory reads and survive Mem- bridges [INFERRED]. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#liveness-probe]
- Bridge memory decode: for every upstream bridge on the sysfs path (derive them with readlink -f, as egpu-nvidia-load.sh does in linux-egpu-hotplug-boot-orchestration.md), read setpci -s $BR COMMAND.w; bit 0x2 = Memory Space Enable, 0x4 = Bus Master (PCI spec COMMAND register) [INFERRED from PCI spec]. Any bridge with Mem clear -> state BRIDGE_MEM_OFF: report and stop; a failed or non-hex read is UNKNOWN. The probe never writes: the fix, setpci -s $BR COMMAND=0x6:0x6 (value:mask, changes only those bits, as egpu-nvidia.service does at boot), belongs to the recovery path behind the gates. Writing a bridge COMMAND register on a running system is [UNVERIFIED] as safe here: that sibling does it at attach time, and its quoted setpci(8) text warns that misuse can hang the machine. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#liveness-probe]
- BAR0 chip ID (opt-in: the template below runs it only when EGPU_BAR0_READ=1 is set in /etc/default/egpu-watchdog; default off, because it is the one probe step that touches device memory space): only if step 3 passed and the GPU's own COMMAND has Mem set, read the 32-bit word at BAR0 offset 0 (NV_PMC_BOOT_0, chip identification) via resource0. 0xffffffff -> GPU_DEAD_MMIO; a plausible non-FF value -> ALIVE. The register name/offset is [INFERRED from NVIDIA open-gpu-doc/nouveau knowledge; not verified this session, and the exact decode of the Blackwell value is [UNVERIFIED]]; use only "not all-ones and stable across two reads" as the criterion. A failed read or an unstable value is UNKNOWN. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#liveness-probe]
What can go wrong: reading BAR0 while the link is dead can stall on completion timeout and raise AER/NMI-class events on some platforms, which is why steps 1-3 gate it and why the template leaves it off by default (with it off, ALIVE means only: function present, vendor ID answers, bridge memory decode on); do not read while a re-init is in progress; reading resource0 while the driver is bound is normally harmless for a single dword but treat it as [UNVERIFIED] on this kernel. A stalled read can leave the probe stuck (an uninterruptible process ignores the unit's timeout); the stale heartbeat is then the alert. setpci config reads may resume a runtime-suspended device (from memory of the kernel's sysfs read path) [UNVERIFIED], so the probe is "no writes", not "no side effects". Output one word (ABSENT|UNREACHABLE|BRIDGE_MEM_OFF|GPU_DEAD_MMIO|ALIVE|UNKNOWN) so alerts and metrics share vocabulary. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#liveness-probe]

Alerting

Metrics path: node_exporter textfile collector: run with --collector.textfile.directory=<dir>; it parses *.prom files in Prometheus text format, timestamps unsupported; write atomically (temp file then mv) [SOURCED https://github.com/prometheus/node_exporter#textfile-collector]. Suggested self-defined metrics (names are this file's own, not a standard): egpu_liveness_state{state="ALIVE"} 1, egpu_pcie_link_speed_gts, egpu_aer_correctable_total, egpu_recovery_attempts_total, egpu_recovery_locked 0|1, egpu_probe_last_run_timestamp_seconds. The probe template emits only egpu_liveness_state and egpu_probe_last_run_timestamp_seconds. Alert on time() - egpu_probe_last_run_timestamp_seconds > 300 or absent(egpu_probe_last_run_timestamp_seconds). [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#alerting]
Push path (no Prometheus): ntfy (HTTP POST to a topic), mail (mailx/msmtp), or a generic webhook via curl. Keep one notify function; rate-limit notifications (one per state change plus a daily reminder) so a flapping card does not spam (the templates send one per state change from the probe and at most one per hour from the alert unit). [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#alerting]
Priorities: page on GPU_DEAD_MMIO, LOCKED_OUT, or probe stale; ticket on BRIDGE_MEM_OFF (auto-fixed only if recovery is enabled) and on a persistent UNKNOWN; log-only on corrected AER below threshold. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#alerting]
Do not alert from inside the recovery path only; send the "attempting recovery" notification before acting so a hang mid-recovery is still visible. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#alerting]

Automated Recovery and Safeguards

Recovery sequence (mirrors the manual egpu-reinit.sh; the order in the "Re-init Without Reboot" section of linux-egpu-hotplug-boot-orchestration.md, which drains and unloads as in its safe-removal runbook, is canonical): stop GPU consumers (Ollama, then nvidia-persistenced) under timeout and confirm nothing holds /dev/nvidia* -> unload modules leaf-first (nvidia_drm, nvidia_modeset, nvidia_uvm, nvidia) -> software hot-remove the enclosure subtree at its top-most bridge (the reason is in linux-egpu-hotplug-boot-orchestration.md; egpu-hot-unplug-pciehp-safety-linux.md removes only the two GPU functions, in its own planned-detach runbook) -> echo 1 > /sys/bus/pci/rescan -> masked Mem+BusMaster write on each bridge on the path (COMMAND=0x6:0x6, the same as that file's 0x0006:0x0006) -> reload modules in dependency order (nvidia, nvidia_uvm, nvidia_modeset, nvidia_drm) and gate on nvidia-smi -L -> start nvidia-persistenced, then Ollama. Abort at the first failure. A busy module means stop: never rmmod -f, never a blind retry (see the lock-holding loop in egpu-hot-unplug-pciehp-safety-linux.md). [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]
Why it is dangerous: remove + rescan is the same operation as a hot-unplug done to yourself; if the card is actually gone or the link is flapping, rescan may not bring it back, and the box loses the GPU until reboot. NVIDIA's action for Xid 79 is a restart [SOURCED analyzing-xid-catalog.html]. Hence the default is alert-only and recovery is opt-in. The cheapest fix (the masked bridge Mem write) is meant to run as tier 1, before the full sequence; the skeleton below omits that tier. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]
UNKNOWN (the probe could not decide) is treated like ABSENT and UNREACHABLE. The diagram is design intent: the templates implement classification and the gate (mode, lockout, backoff, eligibility), not the two-consecutive-probes rule, VERIFY, the COOLDOWN return or the success reset. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]
Gates (all must hold, else stay ALERT_ONLY) [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]
- Mode file says auto (default alert; a missing file means alert), is root-owned and not group- or world-writable. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]
- Probe state is fresh (under 3 minutes) and is BRIDGE_MEM_OFF or GPU_DEAD_MMIO; presence + vendor-ID confirmed at least twice, 30 s apart (not a transient); and no pciehp Link Down in the last 120 s without a matching Link Up (a physically removed or power-cycling enclosure). [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]
- No job mid-flight: no client process with GPU compute (e.g. no active Ollama request; check the Ollama API /api/ps idle state, which as recalled lists loaded models rather than in-flight requests, so it is a weak signal [UNVERIFIED]; confirm with curl -s localhost:11434/api/ps during a request, port from memory; or a maintenance flag file) and no in-progress model load [INFERRED; endpoint behaviour UNVERIFIED here]. Prefer an explicit lock file consumers can take. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]
- No logged-in session depending on the GPU: loginctl list-sessions shows no graphical/seat session using the device, and no process holds /dev/nvidia* other than the allowed consumers (fuser/lsof). On a headless box this is normally true; if not, refuse. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]
- Budget (the numbers are the author's own design choices, not measured or sourced): at least 1 h since the previous attempt, backoff doubling (1h, 2h, 4h), and a hard cap of N=3 unsuccessful attempts. State lives under /var/lib/egpu-watchdog/, survives reboots (stricter than a per-boot cap) and clears only on a verified success or a human unlock. After N failures -> LOCKED_OUT: alert loudly, do nothing else until a human removes the lock. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]
- Not within 10 minutes of boot (a placeholder value) or while egpu-nvidia.service is activating (avoid racing the loader). [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]
- Single instance: flock held by egpu-recover.service for the whole run, and the same lock taken by egpu-reinit.sh so a manual run and the service exclude each other. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]
- Never reboot automatically (see Anti-patterns). Optionally, on LOCKED_OUT, raise "reboot needed" for a human. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]
Skeleton coverage. egpu-recover-gate below enforces gate 1 (exit 11), gate 2's state and Link Down checks (20, 16; any Link Down in the window refuses, as does an unreadable journal), gate 3's JOB_ACTIVE file (15), gate 4's fuser check (17), gate 5 (13, 14, 22), gate 6 (18) and the "attempting recovery" notification (the "after" message is the reinit script's job). It omits gate 2's 30 s repeat, the Ollama check, loginctl, the counter reset after a success, gate 7 (the service holds the lock; adding it to egpu-reinit.sh is yours) and any reboot logic. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]
Break-glass (manual): run egpu-reinit.sh by hand from a real console or SSH after reading the alert and confirming the card is present (lspci -D -d 10de:); the gate does not run on that path, so you own the checks. sudo touch /etc/egpu-watchdog.disable stops all mutation (exit 10). After LOCKED_OUT, unlock by hand: sudo rm /var/lib/egpu-watchdog/LOCKED_OUT && echo 0 | sudo tee /var/lib/egpu-watchdog/fails. A --unlock flag and a FORCE override are not in the skeleton; a FORCE should skip only the backoff, never another gate. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]
Switching modes deliberately. Alert-only is the default: with no /etc/egpu-watchdog.mode, or content other than exactly auto, the gate refuses (exit 11) and nothing mutates. To enable recovery, in order: (1) run egpu-reinit.sh by hand once, in a maintenance window with console or SSH access; (2) refine the fuser gate and test the gate against mocked states; (3) install egpu-recover.service and start it by hand only; (4) write the success handler the skeleton omits (reset fails to 0 after a verified recovery; otherwise three successful recoveries lock the box out); (5) echo auto | sudo tee /etc/egpu-watchdog.mode (root-owned, 0644). Revert with echo alert | sudo tee /etc/egpu-watchdog.mode, or hard-stop with the disable file. Running recovery on probe failure (an OnFailure= drop-in on egpu-probe.service) is a separate, later decision; backoff and lockout still bound every run. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]
What could go wrong: consumer stop hangs (use timeout, then abort, do not continue to unload); module unload fails because a process still holds the device (abort and alert; never rmmod -f); rescan brings the card back at a different bus address than the loader expects (re-derive the path each time); re-init succeeds but Ollama restarts into a half-alive GPU (VERIFY state runs the probe three times and a tiny nvidia-smi query before declaring success). [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#automated-recovery-and-safeguards]

Templates

All templates are alert-only by default. Adjust the BDF/bridge values; paths are under /usr/local/sbin and /etc/systemd/system. Untested on this box. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#templates]
Everything here runs as root (units do by default; use sudo in a shell). File modes: scripts 0755, units 0644, root-owned; /etc/default/egpu-watchdog 0600 (it can hold NTFY_URL, and an ntfy topic name works as a shared secret [UNVERIFIED]); EnvironmentFile= lines are plain KEY=value, quoted when they contain spaces; set EGPU_BDF, EGPU_BRIDGES and NTFY_URL there instead of editing the script. Install from the directory where you saved the files below (this leaves out the opt-in recovery pieces on purpose and does not overwrite an existing environment file): [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#templates]
/usr/local/sbin/egpu-probe (reads only; exits 0 when ALIVE, 1 otherwise; writes textfile metrics and notifies on state change) [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#templates]
What could go wrong: with EGPU_BAR0_READ unset or 0, ALIVE means only "function present, vendor ID answers, bridge memory decode on" and the probe cannot report GPU_DEAD_MMIO. Turn the read on only after watching it on this kernel: run sudo EGPU_BAR0_READ=1 /usr/local/sbin/egpu-probe by hand while the eGPU is healthy and idle, confirm it prints nothing bad and the state stays ALIVE, then set it in the env file. The read takes about 1 s more per run, and a read stuck in the kernel ignores the 5 s timeout, so a stale heartbeat is the alert. The read is [UNVERIFIED] as harmless while the NVIDIA driver is bound. A failed setpci gives UNKNOWN, not a verdict. The GPU address is hard-coded and a moved one reads as ABSENT, which only alerts; the bridge list is derived from the sysfs path each run (EGPU_BRIDGES overrides it, and a stale override can misreport BRIDGE_MEM_OFF, so avoid it). setpci needs the pciutils package. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#templates]
/etc/systemd/system/egpu-probe.service (alert-only; failure of the probe triggers the notifier) [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#templates]
The disable file is deliberately not a ConditionPathExists= here: it stops mutation only, and a skipped probe would page as a stale heartbeat. Pause monitoring by stopping egpu-probe.timer. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#templates]
/etc/systemd/system/egpu-probe.timer [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#templates]
/etc/systemd/system/egpu-alert@.service (notification only; no recovery; at most one alert per hour [INFERRED author's choice], because the failing probe triggers this unit every minute; $$ passes a literal $ through systemd to the shell) [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#templates]
Enable: sudo systemctl daemon-reload && sudo systemctl enable --now egpu-probe.timer. Verify: systemctl list-timers egpu-probe.timer, sudo journalctl -t egpu-probe, and cat /var/lib/node_exporter/textfile/egpu.prom. Test the alert path harmlessly with sudo systemctl start user@example.test. OnFailure= and start-limit semantics per [SOURCED https://man7.org/linux/man-pages/man5/systemd.unit.5.html]. The OnFailure= chain has not been run on this box, and its systemd version was not recorded (systemctl --version) [UNVERIFIED]. Whether Type=oneshot plus a non-zero exit is treated as failure on this systemd version: the man page says it is; confirm with sudo systemctl start egpu-probe.service; systemctl is-failed egpu-probe.service when the card is deliberately absent (in a maintenance window). [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#templates]
Opt-in recovery: NOT installed and NOT enabled by default. Install neither piece (same install modes as above: 0755 for the gate, 0644 for the service) until you have followed "Switching modes deliberately" above. The service holds the lock and runs the gate, then your egpu-reinit.sh; the gate only checks. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#templates]
/etc/systemd/system/egpu-recover.service (no [Install] and no timer: only a person, or an OnFailure= drop-in you add after testing, starts it) [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#templates]
SuccessExitStatus= lists the gate's refusals so they do not alert; exit 13 (locked out) and 22 (bad state) still do. The wrapper maps every egpu-reinit.sh failure to exit 1, so a reinit exit code that happens to be in that list cannot pass as a refusal. Keep the default start limit and add no Restart=. TimeoutStartSec=300 is the author's own choice [INFERRED], and a timeout kill can leave a half-finished recovery. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#templates]
/usr/local/sbin/egpu-recover-gate (skeleton; the mutating body is deliberately your existing egpu-reinit.sh) [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#templates]
What could go wrong: the gate counts the attempt before acting and refuses when its state files are unreadable or unwritable, so a hang or a full disk cannot cause an unbounded retry. The lock sits in the service because one taken inside the gate would drop when the gate exits, before the mutating body runs. The fuser gate refuses whenever Ollama or persistenced hold the device, which on a healthy box is always; refine it to allow exactly those two (your reinit script sequences the consumer stop) before enabling auto, never "any holder". JOB_ACTIVE only works if consumers create it. Test against a mock GPU state (a copy of the gate pointed at a scratch state directory) before ever setting mode to auto. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#templates]

Anti-patterns

Auto-reboot loops: this design never reboots on its own. Rebooting on Xid 79 without a boot counter turns a bad cable into an endless boot-fail loop; if you add a reboot anyway, cap it at once per N hours and persist the counter across boots. Prefer alert-and-wait. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#anti-patterns]
Unbounded retries / no backoff: recovery every minute hammers a marginal link and can wedge the Thunderbolt controller. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#anti-patterns]
Recovering during removal: acting on pciehp: Link Down with remove/rescan while the card is being unplugged or the enclosure is power-cycling (see egpu-hot-unplug-pciehp-safety-linux.md). [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#anti-patterns]
Acting on Xid 154 alone, or on a single log line: 154 is a summary [SOURCED analyzing-xid-catalog.html]. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#anti-patterns]
Calling nvidia-smi without timeout from the watchdog: the monitor itself hangs when the GPU is wedged. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#anti-patterns]
rmmod -f or killing consumers with SIGKILL to force progress. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#anti-patterns]
Monitoring only the log: counters and state (sysfs, liveness) catch silent downtraining and Mem- bridges that log nothing. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#anti-patterns]
No heartbeat: a dead timer looks like a healthy GPU. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#anti-patterns]
Losing evidence: volatile journal wipes the pre-failure log on reboot; set Storage=persistent (confirm the spelling, see Log Signatures). Also snapshot dmesg/lspci -vv into a root-only file before any recovery step (the gate skeleton does not). [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#anti-patterns]
Writing config space blindly (e.g. resetting the whole COMMAND register): in the recovery path only, behind the gates, set just the Mem and Bus Master bits in value:mask form (COMMAND=0x6:0x6). The probe never writes. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#anti-patterns]
Reporting a probe error as a verdict: a failed setpci or read gives UNKNOWN, never BRIDGE_MEM_OFF or GPU_DEAD_MMIO, which can open the recovery gate. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#anti-patterns]
Locking only around the checks: the lock drops when the checker exits, before the mutating body runs; take it in the caller. [INFERRED] [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#anti-patterns]
Silent auto-fix: always notify before and after any mutating action. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#anti-patterns]
Treating soft re-init as a cure for power faults: a browning-out enclosure PSU produces the same signatures; see egpu-power-enclosure-and-thermals-linux.md. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#anti-patterns]

Related references added later: egpu-unattended-remote-recovery-and-out-of-band-linux.md (host-first escalation ladder, remote power control and out-of-band access); egpu-reproducible-bringup-and-drift-detection-linux.md (capturing this wiring as a restorable manifest, drift verifier and restore order). [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#anti-patterns]

Sources

Kernel PCIe AER guide (log format, sysfs counters pointer): https://docs.kernel.org/PCI/pcieaer-howto.html [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sources]
Kernel ABI sysfs-bus-pci (power_state, d3cold_allowed, reset, remove): https://raw.githubusercontent.com/torvalds/linux/master/Documentation/ABI/testing/sysfs-bus-pci [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sources]
Kernel ABI sysfs-devices-power (runtime_status values): https://raw.githubusercontent.com/torvalds/linux/master/Documentation/ABI/testing/sysfs-devices-power [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sources]
NVIDIA Xid catalog (79, 119, 120, 154 and actions): https://docs.nvidia.com/deploy/xid-errors/analyzing-xid-catalog.html [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sources]
systemd.unit(5) (OnFailure, StartLimit*, ConditionPathExists): https://man7.org/linux/man-pages/man5/systemd.unit.5.html [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sources]
Prometheus node_exporter textfile collector: https://github.com/prometheus/node_exporter#textfile-collector [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sources]
nvidia_gpu_exporter: https://github.com/utkuozdemir/nvidia_gpu_exporter [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sources]
NVIDIA dcgm-exporter: https://github.com/NVIDIA/dcgm-exporter [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sources]

Not fetched (two man-page hosts returned 403 during research): journald.conf(5), journalctl(1), NVML API reference, egpu-community watchdog scripts. Statements resting on those are tagged [INFERRED]/[UNVERIFIED]. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#sources]

Unverified items (summary)

Exact AER sysfs attribute names/format (ABI aer_stats path 404'd); confirm with ls /sys/bus/pci/devices/<BDF>/ | grep aer. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#unverified-items-summary]
current_link_speed/current_link_width presence in ABI text (not shown in fetched page); confirm with cat /sys/bus/pci/devices/<BDF>/current_link_speed. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#unverified-items-summary]
BAR0 offset 0 as chip-ID register and its Blackwell value decode; safety of reading resource0 while bound (why the BAR0 read is off by default); safety of writing a bridge COMMAND register on a running system; whether setpci config reads resume a runtime-suspended device. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#unverified-items-summary]
DCGM field support on GeForce; nvidia_gpu_exporter native-backend Xid counters on driver 610.x. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#unverified-items-summary]
journald Storage=persistent spelling and journalctl -g availability (from memory); confirm with man journald.conf and journalctl --help. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#unverified-items-summary]
OnFailure= re-triggering on each failing timer run, default start-limit numbers, systemd version (systemctl --version). [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#unverified-items-summary]
Ollama idle-detection endpoint behaviour (/api/ps as a loaded-model list, default port); confirm with curl -s localhost:11434/api/ps during a request. [archived source: https://llms-explorer.com/sources/global-ai-hub/egpu-health-monitoring-and-automated-recovery-linux/#unverified-items-summary]

Children

AER sysfs counters and rate alerting (frontier)
BAR0 chip-ID liveness probe (frontier)
DCGM vs NVML vs nvidia-smi on GeForce (frontier)
NVIDIA Xid recovery-action taxonomy (frontier)
PCI bridge COMMAND Mem/BusMaster diagnosis (frontier)
auto-recovery rate limit, backoff and lockout state machine (frontier)
break-glass manual recovery runbook (frontier)
node_exporter textfile collector for hardware health (frontier)
persistent journal kernel-evidence retention (frontier)
systemd OnFailure oneshot watchdog pattern (frontier)

Frontier under this node: AER sysfs counters and rate alerting, BAR0 chip-ID liveness probe, DCGM vs NVML vs nvidia-smi on GeForce, NVIDIA Xid recovery-action taxonomy, PCI bridge COMMAND Mem/BusMaster diagnosis, auto-recovery rate limit, backoff and lockout state machine, break-glass manual recovery runbook, node_exporter textfile collector for hardware health, persistent journal kernel-evidence retention, systemd OnFailure oneshot watchdog pattern

← the whole tree · 3D view· how to read this page