Auditing our own systems: the check that ran for twenty-three days and could never have caught anything

The problem

Every business runs on systems that report on their own health. The backup that emails a green tick. The alert that says nothing because nothing is wrong. The monthly report that arrives on the first, correct in shape and never once questioned.

The hard thing about all of them is that “nothing found” and “not looking” produce the same output. A quiet monitor and a broken monitor are the same silence. You usually learn which one you had from a customer.

So the role is not “watch the dashboard.” The role is: check whether the thing that watches the dashboard is real. That is a job, it is dull, it is nobody’s favourite afternoon, and almost no small business has anyone doing it.

We ran it on ourselves. It found two faults in the same mechanism, in the same week, and they are opposite failures — one where the check could never speak, and one where it spoke and was wrong. The second is its own case study. This is the first.

Both were found here rather than by anything reporting them. Each one is below.

The approach

The rule the audit was run under is one sentence: check that a mechanism arrived, not that it fired.

Firing is easy to verify and proves nothing useful. A scheduled job that exits zero has fired. A log line saying the check completed is a true statement about the check completing. Neither of those is evidence that anything was examined, and both read exactly like coverage — which is worse than a gap you can see, because a visible gap gets fixed.

Arriving means the thing at the far end of the chain actually received something and acted on it. The way you test that is to go around the outside and re-derive the answer independently. If the system says all clear and an outside pass says the same, that is a check. If the system’s own log is the only evidence, that is a story.

The monitor that could never speak

It ran on every stop for twenty-three days. Its records were still empty.

The check exists for a real, specific failure. One of our agents once ran 126 minutes on a job estimated at 43, and was never silent for a single minute — it wrote to its log every few seconds the whole time. Every liveness check said healthy, and every one of them was right. It was healthy. It was also an hour and twenty past due, and nothing in the system could see that, because being alive and being on time are different questions and only one of them was being asked.

So a second check was built to ask the other one, and wired to run on every stop from 6 August.

It ran. It ran hundreds of times. It never once had anything to read.

The check looks for jobs past their expected finish time. The expected finish time is written by exactly one other program — and that program had “eta”: None hardcoded in all five of the places it writes. Not left out, not occasionally missing. Written as nothing, five times, deliberately-looking, on every path through the file.

The check was therefore reading a list of jobs of which precisely zero could ever be judged late, and correctly concluding that nothing was late. Its own state file — the record of everything it had ever warned about — was still empty after twenty-three days, and an empty file is exactly what a quiet system looks like.

Nothing about this was visible from the inside. The scheduler logged success, because it succeeded. The check exited zero, because it was working. Our own written standard described the estimate as being written at dispatch, and that sentence had been false since the day it was written.

It was found by re-deriving the answer from outside: taking the raw dispatch records, pairing them against the real end times on disk, and getting a number where the system had a blank.

The second break in the same chain

Fixing the first fault alone would have looked exactly like a fix and changed nothing.

The obvious repair is to make the estimate real. That was done — and it would not have worked.

Reading further along the same chain turned up a second, completely independent break. The check only considered a job if its status read running. The one program that writes those statuses has always written working.

So even with a correct estimate written into every job, the line that compares it against the clock could never have been reached. Two faults, in two different files, written by different hands at different times, each one on its own sufficient to render the whole mechanism silent.

This is the part worth taking away, and it is not really about monitoring. A fix that addresses the fault you found, in a chain you have not walked end to end, produces a system that is still broken and now believed. The first repair would have shipped, been announced, logged success, and delivered nothing — which is the identical failure it was repairing, one layer along.

The estimate is no longer declared, either. It is measured: 540 completed jobs paired against their real end times, and the ninetieth percentile taken per person. That last choice matters more than it sounds. Half of all jobs run past the median, so a median deadline fires on half of everything and is ignored inside a week — which is how the original check would have died even if it had worked. And a single number for everybody was never going to hold: one specialist’s typical job takes a minute and a half, another’s takes twenty-nine.