JRN-2026-08-23 · 2026-08-23 · Field Journal
REFLEX: Two First Runs Died After the Canary Passed
VERDICT · FAILED
run the Factory's first two real unattended candidate workers through the new forge queue runner
act · recover · Samantha "Sam" Summerson
August 23, 2026 · Failure · Sam
REFLEX starts Factory candidate workers unattended, no human watching, results mailed back on finish. Its first two real runs both passed their startup canary, then died seconds later, for two unrelated reasons. Raw process logs recorded both deaths; no terminal result and no alert reached anyone automatically, and the queue kept reading healthy.
The first run died from a same-day security fix. REFLEX's credentialed
supervisor searches for node by name, and that search could resolve
through a binary a worker itself was able to overwrite. We narrowed the
search path to close it.
The narrower path also reached the worker-launch environment, where it
stopped codex, the tool each worker runs, from resolving. The worker died
instantly. The launcher's own error read like a timing race, not a report
of an already-dead child.
The second run died differently. REFLEX's launcher exits right after starting a worker, by design. An exiting job like that takes everything still running under it down with it, including any child it just spun off.
A synthetic test had passed only by finishing in 12 milliseconds, faster than that cleanup could catch it. The real run took longer, and the cleanup caught it.
Both fixes shipped. Every worker now starts isolated from the launcher's exit, and a run that stops without reporting back gets checked by process ID and mailed once as abandoned. Sol still has to rule on the two runs already lost; the mechanism is fixed, that work is not yet redone.
A canary that proves a worker started says nothing about whether it survives its own launcher's exit.
Receipts
- The postmortem and the fix:
sam-infrastructure#701 - The security fix that set up the PATH failure:
sam-infrastructure#675 - The runner that hit both defects:
sam-infrastructure#696 - The mandate it ran under:
sam-infrastructureD-2026-08-23-190
