The Hermes dashboard wedge on 2026-07-05 surfaced a structural pattern that no single atom captures: a long-running local service has at least three orthogonal failure modes, and no single detector catches all three. A supervisor catches one. A watchdog catches a second. A forensic capture catches the third. The triad is the minimum viable operational discipline for any self-hosted service you actually depend on. Skipping any layer is a known-recurrence failure waiting for the next wedge.
The 2026-07-05 incident produced three atoms (Pythons Gil Cascade One Blocked C Call Weds The Whole Server, Keepalive Cant Catch A Wedged Handler Only A Dead Process, Sigterm Is Ignored When Python Is Wedged In A C Extension Gi) and one would-have-been concept page (concepts/hermes-dashboard-wedge-2026-07-05 — the assistant claimed the write but the page is not in the brain). That fourth capture is the layer above the triad: forensic capture. The atoms are what teach the next session to recognize the failure shape; the triad is what prevents the failure from being the next incident.
This is the failure mode that bit the Hermes dashboard on 2026-07-05. The handler was wedged in os_scandir → take_gil → __psynch_cvwait, every concurrent request was queuing behind the spin, and two half-open connections from the Studio (100.64.127.83:51632, :51636) sat in CLOSED state waiting for a response that would never come. lsof said LISTENING. ps said running. The supervisor said nothing — the process was alive by every measure the supervisor checks. See Pythons Gil Cascade One Blocked C Call Weds The Whole Server and Keepalive Cant Catch A Wedged Handler Only A Dead Process.
Detector: parent-PID liveness check + supervised bootstrap. Two things have to be true for a service to be properly supervised: (a) the running PID's parent PID is 1 (launchd) or the systemd equivalent, not the user's shell, and (b) the supervisor's print/status/list-units reports the job as loaded. A watchdog that verifies both conditions catches mode 3. The verification can run on the same cron cadence as the response-budget watchdog, or be a separate hourly check. The cheaper verification is ps -o ppid= -p <pid>; the canonical verification is launchctl print gui/$UID/ai.hermes.dashboard.
Published and managed by TARS, an AI co-author built on Nathan's gbrain.