The Hourly Job That Stopped Firing and Nobody Noticed — Until the Watchdog Barked

A scheduled CI validation job is the quietest kind of infrastructure. It runs on the hour, every hour, and when it works you never think about it. That is exactly why a silent miss is so dangerous: a job that does not run produces no red status. It just stops being true that your main branch is validated, and nothing in the world tells you so. That is what happened to us. Our backend validation window — the only continuous check on main-tip, and the only CI path exercising four Node-gated JavaScript modules — silently stopped firing. The preceding runs had come every 58 to 61 minutes, then nothing. No failure, no red, no alert. Just a gap where a guarantee used to be. A missing schedule looks identical to a healthy one unless you are watching the clock, and nobody was. The fix was not to make the job run more reliably — it was to make silence impossible to mistake for health. We built a stale-schedule alert based on elapsed time since the last observed run, not on the firing minute. If the latest run is older than the limit, it warns. Silence can no longer masquerade as green. And then the alert did the one thing that separates a real watchdog from a hypothetical one: it fired. Hours after the fix landed, the schedule slipped again — a commit landed in the same second the hourly window was due, Gitea re-registered the schedule, and the fire went with it. At 15:17:43Z the alert caught it: the latest scheduled backend run was 77.1 minutes old against a 75-minute limit. The miss that would once have been invisible was now a named, timestamped warning on the board. The lesson is not that our scheduler is fragile. It is that the expensive failure mode in automation is not the crash — it is the quiet stop, the job that dies without telling anyone. If you operate anything on a schedule, spend the effort on detecting the silence, not just on hoping the schedule holds. The watchdog that barks once at a real miss is worth more than a hundred green runs nobody looked at.