The overdue alarm that was wrong three times in two days

The problem

Somebody in a business watches the board and says when a thing is stuck. It is one of the easiest roles to describe and one of the hardest to keep worth having, because the failure is not usually that it misses something. The failure is that it is wrong often enough to stop being read.

Everyone has met the version of this that has already died. The inbox rule that files 400 alerts a week. The dashboard with a red panel that has been red since March. The monitoring tool somebody bought, configured on a Tuesday, and now closes without looking.

None of those were switched off. They were all believed, once. A checker that cries wolf trains the person watching it to skim, and the training is permanent — the alert that finally matters arrives into a habit of not reading, and there is no way to signal that this one is different.

Ours got three days into being useful and started doing exactly that.

Three false alarms in two days. Two had one cause and the third could not have been fixed by tuning. Each is below.

The approach

The measure of this fix is not that the alarm became accurate. It is that it became quiet.

That distinction drove the whole repair. Every change was tested by asking what it now stays silent about, rather than what it now catches — because the thing being protected is somebody’s willingness to read the next one, and that is spent by false alarms and not replenished by true ones.

It also reports and never intervenes, deliberately. It cannot stop a job, extend a job, or kill one. Running long is not the same as being wrong — work legitimately overruns, and a monitor that acted on a schedule would destroy good work on the strength of an estimate. It says the thing is late. What to do about that stays with a person.

Reading a two-hour-old snapshot as if it were now

Every field it read was accurate. None of them was current.

At 15:33 the alarm reported a job 1h 49m over. That job had been handed back complete at 13:06, thirty-eight minutes ahead of its estimate, and the work acting on it had landed a minute later. Nothing was late. Nothing was even running.

The monitor reads a snapshot file describing who is working on what. That file records, per job, a status and how long it has been quiet. The row it read said working, and quiet for about one minute.

Both were true. Both were true at 13:07, when the snapshot was written.

The order of operations is what did it: the program that writes the snapshot runs after the monitor in the same sequence, so the monitor is always reading the previous cycle’s file — and across a gap in activity, that file can be hours old. A stale snapshot does not gently become less accurate. It lies with confidence, in one specific direction: a job frozen at “working” stays “working” forever, while its expected finish time recedes further into the past every minute. It is guaranteed to eventually produce a false alarm, and never a false all-clear.

The snapshot had been stamping itself with its own write time the entire time, and nothing had ever read that field. The fix adds the file’s own age to everything it claims: a fresh snapshot is unchanged, and a two-hour-old one puts every job it describes far past the point where it could be describing anything current — which is the honest answer, because after two hours that file cannot tell anyone what is running now.

The false alarm that no amount of tuning could catch

At the instant a job finishes, “just finished” and “still going” are the same reading.

The third one is the interesting one, because it survived the fix above and would have survived any refinement of it.

The guard against false alarms was based on silence: a running job writes constantly, so a job that has gone quiet has probably stopped, and quiet jobs are not judged late. That works, and it is the right idea for a job that has gone quiet.

It cannot work for a job that writes right up to the second it hands back. At that instant its silence measures approximately zero — identical to a job in full flow. The monitor fired on one specialist twice inside twenty minutes, and the second time the record of its completion had been written in the same minute the warning went out.

No threshold fixes this. There is no number of seconds that separates the two cases, because there has not been any silence yet. A heuristic about quiet cannot distinguish “stopped a moment ago” from “still producing” — it is being asked a question it has no information about.

Something else did know. A separate record was already being written every time a job closed: a real timestamp, not an inference about liveness, saying that the thing stopped. It had been on disk the whole time and nothing read it.

The monitor now reads it, and one detail in how it reads it is the difference between a fix and a new bug. It does not ask “has this person ever completed something” — that would silence the alarm permanently after anybody’s first job of the day, and people here are re-assigned constantly. It asks whether their most recent record is a closing one. That is the actual question: not “have they ever finished”, but “is anything open for them right now.”