Failure Analysis
•
2026-07-30
The Relay Was Live. The Lock Order Was Wrong.
The failure was not a bad filter or a dropped event. It was a lock-ordering problem in a relay subscription registry: one path held an index guard while looking up authoritative subscription state, while another held that authoritative state while cleaning the index. Under the right interleaving, both paths could wait for the other and park the runtime workers that needed to make progress.
That distinction matters for decentralized systems. A relay is not dependable merely because it is reachable, and a subscription is not correctly delivered merely because the matching logic passes its unit tests. Liveness, routing scope, and cleanup order are separate contracts.
The cycle lived between two correct-looking paths
Buzz's subscription registry keeps authoritative subscription state in `subs` and maintains indexes to make fan-out sub-linear. The original fan-out path held an index entry while it called `push_match`, which then acquired `subs` to inspect the subscription. Close and same-ID replacement took the reverse route: acquire `subs`, remove the subscription, then mutate the index.
Each local operation looked reasonable. The composition was not. Fan-out could wait on `subs` while cleanup waited on an index guard. That AB/BA cycle was reachable during concurrent delivery and replacement, and a synchronous wait inside the relay could stall every Tokio worker that encountered the cycle.
The correction changed guard lifetimes
The fix made the order explicit. Removal now takes the authoritative `subs` entry, removes the subscription, cleans the indexes, and only then releases the `subs` guard. Fan-out does the inverse work safely: it clones each bounded candidate vector, releases the index guard, and only then performs the authoritative lookup.
Snapshotting does not make a candidate authoritative. A subscription can be replaced after the snapshot is taken. `push_match` therefore re-checks the community and channel against the current subscription before returning a match. The result is a small but important two-step rule: release the index guard before the authoritative lookup, then validate scope against authoritative state.
The regression tests describe the failure modes
The new coverage does more than assert that a normal fan-out returns the expected connection. One test pauses old index cleanup and proves that a same-ID replacement cannot finish early or disappear from the index. Another captures a stale candidate snapshot, moves the subscription to a different channel, and verifies that the old event does not cross that scope boundary.
A concurrent fan-out and replacement test then runs both operations repeatedly against the same registry with a bounded deadline. The test is a liveness guard: the threads must complete. It is not a production load test, but it makes the intended concurrency contract executable.
What the proof covers—and what it leaves open
The Buzz change was validated with the pinned Rust 1.95 relay suite: 769 library tests passed, 33 were ignored, 11 binary tests passed, and doc tests passed. The repository push hooks and `git diff --check` also passed. Those results support the claim that the corrected code and regression coverage are coherent in the checked suite.
They do not prove that every production wedge had this cause, nor that the relay has no other liveness failure. The correction also pays a bounded allocation cost by cloning candidate vectors before matching. That is a deliberate trade: a small copy cost is easier to reason about than nested concurrent guards, but it still deserves measurement as indexed candidate sets grow.
A reusable review question
When a shared registry has an authoritative map plus one or more indexes, draw the lock graph before optimizing the lookup. Name which structure is authoritative, define one acquisition order, and decide where snapshots become stale. Then test three things separately: the operation finishes, the result remains in the right scope, and cleanup does not erase a replacement.
This is the kind of boundary work that tends to disappear behind a green status screen. It is also where community-owned infrastructure earns trust: not by claiming that partial failure is solved, but by making the dangerous interleavings visible enough to test and revise.
Field Notes Used
Buzz · commit 22be8bb35 · PR #3413
Relay subscription lock inversion fix: consistent cleanup order, candidate snapshots, authoritative scope rechecks, and concurrent fan-out/replacement regression coverage.
Validation receipt · Rust 1.95 relay suite
769 library tests passed, 33 ignored, 11 binary tests passed, doc tests passed; push-hook suites and git diff --check also passed.
Next Step
Need implementation support for this topic? Resonant Constructs offers consultation services for practical AI adoption, private knowledge systems, local deployment, and human-reviewed automation.
Archive Note
This Insights surface currently uses a structured in-repo content model. As publishing expands, tag routing will continue mapping each topic to the most relevant consulting landing page.