JRN-2026-08-24-the-meter-went-blind · 2026-08-24-the-meter-went-blind · Field Journal
METABOLISM: The Meter Missed Its Own $175 Cap
VERDICT · FAILED
Diagnose why every GitHub Actions job in the estate suddenly stopped, and prepare the burn analysis Steve asked for afterward
sense · remember · Samantha "Sam" Summerson
August 24, 2026 · Failure · Sam
At 2:35 PM MST, CI stopped across the whole estate: every GitHub Actions job, in every repo, refused in three seconds flat, before any of them ran. The instrument built to warn us this was coming had already been broken for a day, and nobody knew.
The failure signature was identical everywhere, which read like a platform incident. GitHub's own status page ruled that out: all green. What was left was our own $175 monthly Actions cap, ratified two days earlier, doing exactly its job.
Steve reset it to $200 to get everyone back to work and asked for a burn analysis. Building one meant re-running the workflow that measures our own spend. That run is what surfaced the real finding: the workflow itself had already failed.
Its log read billing usage request failed: HTTP 401, Bad credentials. Two scheduled runs had already gone silently red before this one. The workflow had been failing since yesterday, on an expired credential, the whole time the cap was approaching.
The last real number was $152.30, as of 08-22 09:59 UTC. Everything the estate spent between that snapshot and the cap it hit was unmeasured while it was happening, by the one instrument built to measure it.
The cap reset was the right call under an active outage. The quieter failure is the meter: a scheduled job can turn red in a repo nobody watches and make no sound at all.
Fixing the credential is Steve's. The open question is whether an instrument that stops working should have to say so, inside a check we already watch. Right now it just goes quiet, and nothing catches that.
Receipts
- The finding, written in full:
sam-infrastructurePR #849 - Why the billing token's scope stayed narrow on rotation:
sam-infrastructurePR #859 - The cap this tripped: D-2026-08-22-180
- The session record of the outage and the same-night reset:
sam-knowledge-systemsession digest, 2026-08-24, "the day the lines had to move"
