Everything below was measured on my own machines in October 2026: a Mac mini running a shared Perplexity research runner (browser automation over CDP, port 9226) and a scheduling side on a laptop. The Perplexity page fields I cite will rot when their UI changes. The failure model and the evidence rules won't.
For four days my health checks said the channel was fine, and they were mostly right. The channel was fine. What had been dying was the work flowing through it. On the afternoon of October 28, the shared research runner logged sixteen uncaught-exception failures in two clusters, each with an empty note field and no error text anywhere on disk. The account had quota. The page loaded. The evidence simply did not exist.
This post is the post-mortem: the three-layer model of what my checks actually covered, the per-day numbers, root causes ranked by how sure I am, and the two code changes that make the next death explainable. It is the sibling of an earlier post about the job that ran on time and did nothing. That one was a single job with no one reading its log. This one is a shared channel where every layer's health check passed while the pipeline was dead.
What the three layers actually covered
One shared runner script drives a logged-in browser to run deep research. Three scheduled jobs plus ad-hoc research calls all funnel through it, serialized by a single file lock. Health is policed by a daily verify job that opens the site and checks the page structure. Three layers, each checking itself:
| Layer | What it checks | What it cannot see |
|---|---|---|
| Page/quota (daily verify, 09:10) | Does the page render, is the sign-in chip present | Whether logged-in state has rotted — its pass criterion is literally "not signed in → OK" |
| Task (runner + file lock) | Exits 4, 12, 13, 14, each with a named reason in the log | Exit 1: an uncaught exception — a code the script never defines; it is Node's default |
| Scheduling (jobs + shared ledger) | Per-day call counts in a shared JSON Lines file | Which job issued a given call — the ledger had no owner field |
The verify job ran green on October 27 and 28 even though the account's login state was gone on both days: it printed login:false and still returned OK, because a page that renders unauthenticated is, to that check, a healthy page. All four documented exit codes carry named reasons: CDP unreachable, lock wait or quota, not logged in, output too short. Code 1 carries nothing: the ledger recorded it with an empty note, no screenshot, no stderr capture.
Two properties that read like small log-hygiene details are the whole bug: failures left no body, and no ledger row said whose job it was.
The channel never went down. It stopped leaving bodies.
My ledger, per day:
| Day | Calls | Successes | Dominant failure | What I can prove |
|---|---|---|---|---|
| Oct 25 | 145 | 25 | 84x exit 12 (lock wait) | A lock-storm day; most callers gave up after the wait window |
| Oct 26 | 28 | 11 | 16x exit 12 | Load cooled, same pattern |
| Oct 27 | 37 | 22 | 8x exit 4 | CDP connection lost; one deep run finished at 11:41 only via a manual re-run |
| Oct 28 | 26 | 3 | 16x exit 1, clustered | 5 failures at 15:07, 7 at 18:06–18:07; every note empty |
| Oct 29 | 1 | 1 | — | By 00:17, two of the three jobs had been disabled, no record of who or why |
The last successful deep run on October 28 was 13:38. After that the channel went quiet: one call on the 29th, succeeded at 02:24, then nothing. Two checks that would have caught the rot early cost almost nothing. Tailing the ledger and counting empty-note exit-1 rows per day — five minutes, no new infrastructure.
Every root cause I could prove was about missing evidence
- Confirmed — failures left no evidence. The exit-1 rows had empty note fields, no stderr on disk, no screenshot. This is the exact mechanism behind the complaint that started the investigation: "the jobs stopped working, and I can't say why."
- Confirmed — the shared ledger has no owner. 145 calls on the 25th, 26 on the 28th: I cannot say which job issued any of them, so I cannot even say whose backlog died.
- Confirmed — the verify job's blind spot is structural. It failed on October 29 only because my laptop's network dropped for six minutes around 10:02 and its 3600-second probe timed out. It sailed through two days of lost login state. It alarmed when it shouldn't and stayed silent when it should have. Both directions, same root cause.
- Inferred, not confirmed — the retirement mechanism. A done marker for one job had existed since the 27th at 11:41, yet its retry job kept spinning on 30-minute cycles. Two jobs were disabled at 00:17 with no reason recorded. I cannot separate "the gate failed" from "a different consumer was issuing those calls" — which is exactly the consequence of root cause 2. So I stopped calling it a root cause.
The ranking is the deliverable. Three confirmed, one explicitly marked inference. A claim I cannot ground in something on disk is a hypothesis, and the incident write-up writes it down as one.
The fix targets the next death, not this one
-
Catch the uncaught path. The runner now installs process-level handlers for uncaught exceptions and unhandled rejections: on a crash, a one-line summary of the error lands in the ledger's note field instead of the field staying empty. The next exit-1 row reads
uncaught: <message>. Someone will be able to say what died. - Put an owner on every call. The ledger row now records which job issued the call. The next time a channel dies, the backlog that died has a name.
And one decision that is not code: I am not re-enabling any runner. Nothing is queued. Restarting a consumer to keep a channel warm is how the 145-call lock-storm day happened. The channel earns its keep on demand.
A dashboard that cannot distinguish "no work pending" from "the worker is dead" is a green light on a dead channel. The rule I keep from this incident: when a failure cannot be explained with evidence on disk, the defect is in the evidence path — and fixing the path, not guessing the cause, is what ends the incident.
More post-mortems like this one live on the blog; the free tools we run are listed on /tools.
Originally published on my blog: https://javaeye.dev/blog/sixteen-unexplained-failures-zero-evidence/
Top comments (1)
A verify job that checks something only a logged-in session can see, like the account name on a page, would have caught the lost login.