My dev.to Gate Couldn't Measure, and the Suite Called It a Regression

Originally published on hexisteme notes.

On October 23 the daily drift report for my gate suite contained this line (translated from Korean):

check_offer_subject_alignment.py: PASS → REGRESSION  ← a fix you'd made broke again.

That note is the loudest thing the drift watcher can say. It is reserved for the case where something I had already fixed has stopped working. The same gate had printed it on October 18, gone back to PASS on October 21, and printed it again on the 23rd. Before that day's fix, the drift log already held ten verdict changes for this one gate, going back to September 7.

When I opened it, the live data the gate checks was nowhere near the failure it guards against. What had failed was the gate's ability to look. The suite had no verdict for that, so it used the loudest one it had.

The two Stop hooks behind this note are on GitHub under MIT: hexisteme/hard-gate-hooks. They ship with their tests and a read-only scanner that prints what they did on **your* machine, not mine — including the case where it tells you they aren't worth wiring up yet. No email, no signup.*

What the gate checks

The gate is one of 22 scripts in a suite I run over my publishing pipeline. This one checks, against live data, that a link I put in some dev.to posts sits on the posts it belongs on (the ones tagged agents) rather than drifting onto unrelated ones. It reads articles/me/published, then makes one comments?a_id= call per commented article: about 100 live API calls per run. A launchd job runs the suite once a day; on October 23 the run landed at 00:19 UTC.

The gate ends with one of three exit codes:

• 0 ALIGNED: the link sits where it should
• 2 REVERSED: the link has drifted onto the wrong posts, which is a real finding
• 1 UNMEASURABLE: it could not read the live data

Inside the gate those three meanings are distinct. The runner only saw two of them.

The runner knew one kind of non-zero

The suite runner turns each gate's exit code into a verdict. Before the fix, the decision was this (comments mine):

def judge_gate_outcome(exit_code, declared_reason, output="", expect=None):
    if exit_code == 0:
        return STALE_BLOCK if declared_reason else PASS
    if exit_code == 124:
        return CRASHED          # the runner's own timeout
    if not declared_reason:
        return REGRESSION       # any other undeclared non-zero lands here
    if expect and expect not in output:
        return REGRESSION
    return BLOCKED

declared_reason comes from a file, gate_human_blocks.json, where I can declare that a gate is red because it is waiting on a person. A declared gate that fails reads BLOCKED instead of REGRESSION, and a declared gate that passes reads STALE_BLOCK, so exemptions can't outlive their reason. This gate had no declaration. Exit 1 and exit 2 went down the same branch: "I couldn't read dev.to at 00:19" and "the link has drifted onto the wrong posts" both came out as REGRESSION, and the watcher put its broken-fix note on both.

How I worked out that the 00:19 run never measured

The drift log stores verdicts only. It keeps neither the exit code nor the stderr of each scheduled run, so I could not look up what happened at 00:19. I had to reason from a side effect.

The gate caches its last successful live measurement and reuses it for 24 hours. It writes the cache only right after a successful fetch. If loading the API key or the fetch raises an error it expects, such as a network error or a timeout, it prints UNMEASURABLE to stderr and exits 1 before the write. Any other exception in that span also exits 1, without that line, and also before the write.

At 13:32 UTC I ran the gate by hand, without the flag that forces a refetch. It reported its data as 0.0 hours old: it had found no cache young enough to reuse and had fetched everything live. As far as my session logs and the scheduler list show, nothing else had run the gate in between. The daily launchd job is the only scheduler that calls it, and no session of mine had run it since the previous day's scheduled run. That leaves two ways the 00:19 run could have produced a verdict from data:

• It fetched live successfully, whatever it found. Then it would have written a cache about 13 hours old by 13:32, and the afternoon run would have reused it instead of fetching. It didn't.
• It reused an older cache. Any cache still young enough at 00:19 had to come from the previous day's scheduled run, and the verdict is computed from the cached data alone. That data had produced PASS the day before, so reusing it would have produced PASS again, not REGRESSION.

Neither fits. What fits is a run that failed somewhere before the cache write, before it had any data. The fetch from dev.to is the only network step on that path, so the likeliest reading is that the gate could not read dev.to at 00:19. The logs can't tell me which error it was, or whether the gate caught it.

Here is what the afternoon run measured:

• 76.5% (26 of 34) of agents posts carry the link, against 32.0% (32 of 100) of the other posts
• a second, comment-based measure pointed the same way: 58.8% against 34.0%

That is the alignment the gate exists to protect, by a wide margin. At 00:19 the gate never got to look, and the suite reported that as the gate finding a problem.

Why exit 1 couldn't become "unmeasurable"

The obvious patch is one line in the runner: read exit 1 as unmeasurable. That would have broken other gates. Exit codes are a protocol between the gates and the runner, and in this suite 1 already meant something else. check_reader_turn.py, for one, returns EXIT_MISSING_TURN = 1 when it finds a real defect. Remapping 1 suite-wide would have quietly downgraded that gate's real failures. Python also exits 1 on an uncaught exception, so 1 is what a crashing gate produces by default.

The suite already had a convention. It just wasn't written down where the runner could enforce it. Seven other gates declared EXIT_UNMEASURABLE = 3. This gate declared the same constant with a different value, 1, and it was the only one that did.

The fix

Three small changes:

• The runner documents EXIT_UNMEASURABLE = 3 as the suite-wide convention and returns a new verdict, UNMEASURABLE, for exit 3. A human-block declaration does not turn it into BLOCKED. A measurement gap is not a failure waiting on a person, and letting a declaration absorb it would hide a dark gate behind a reason written for something else. Exit 2 still returns REGRESSION, because a gate that found a bad state should still alarm.
• The drift watcher reports a transition into UNMEASURABLE with its own note ("couldn't measure at all; neither a pass nor a regression; check live access and rate limits") and never with the broken-fix note. In both the suite report and the watcher, UNMEASURABLE sorts just below REGRESSION and CRASHED, and the suite report still prints the tail of the gate's output for it, as it does for those two.
• The gate changes its EXIT_UNMEASURABLE from 1 to 3, joining the other seven.

The runner now reads:

EXIT_UNMEASURABLE = 3  # suite-wide: "I could not measure", not "I found a problem"

def judge_gate_outcome(exit_code, declared_reason, output="", expect=None):
    if exit_code == 0:
        return STALE_BLOCK if declared_reason else PASS
    if exit_code == 124:
        return CRASHED
    if exit_code == EXIT_UNMEASURABLE:
        return UNMEASURABLE     # a declared human block does not convert this
    if not declared_reason:
        return REGRESSION
    if expect and expect not in output:
        return REGRESSION
    return BLOCKED

Three tests keep it from drifting back:

• the verdict table gets rows for exit 3 with and without a declared reason, and both must be UNMEASURABLE
• a guard test walks every gate the runner discovers and fails if any of them declares EXIT_UNMEASURABLE with a value other than 3, which is the exact drift that caused this
• a watcher test asserts that a PASS → UNMEASURABLE transition is worded as a measurement problem and carries no broken-fix wording

The guard test is short:

def test_every_gate_declaring_unmeasurable_uses_the_shared_exit_code():
    offenders = []
    for gate in discover_gate_scripts(REPO_TOOLS):
        text = gate.read_text(encoding="utf-8")
        m = re.search(r"^EXIT_UNMEASURABLE\s*=\s*(\d+)", text, re.MULTILINE)
        if m and m.group(1) != "3":
            offenders.append(f"{gate.name}={m.group(1)}")
    assert not offenders

The tools test suite passed, 113 tests. I re-ran the drift watcher. It recorded REGRESSION → PASS for the gate and the false alarm cleared.

A check has three outcomes, not two

A check can look and find everything fine, look and find a problem, or fail to look at all. A runner that only has words for the first two folds the third into one of them.

Fold it into the first and a check that cannot run reports something that looks like a result. In A Measurement That Never Ran Became a Verdict, every read failed and the script still printed a normal-looking verdict built entirely out of failed reads; part of that fix was a guard that refuses to turn a failed read into a zero. In The Scheduled Step Failed on Night One. The Heartbeat Stayed Green., a nightly job's step failed on its first scheduled run while the job's own heartbeat kept reporting healthy.

Fold it into the second and you get what I had: a gate that cries regression on every transient outage. An alarm that is wrong often enough gets read as noise, and a real reversal printed on the same line, with the same note, gets read the same way. The exit code is also where Exit 0, Empty stdout: the Quota Died on stderr goes wrong, from the other side. There, a CLI worker hit its usage cap and exited 0 anyway, so a dispatcher that only checks the exit code would log a completed task. In that case the process sent the wrong code. In my suite the gate sent the code it meant, and the runner read it with the wrong dictionary.

Test runners already draw this line. pytest has a separate exit code for "no tests were collected", distinct from the one for failing tests: a run that tested nothing is not reported as a run where tests failed. A gate suite needs the same vocabulary. The measurement gap gets a verdict loud enough to see, because the gate went dark and someone should check live access, and quiet enough not to pass for a regression.

Applying this to your own gates

• Pick one exit code for "could not measure" and document it in the runner, not only in each gate. Avoid 1: Python exits 1 on an uncaught exception, so a crashing gate would read as unmeasurable.
• Give the runner a verdict for it that is neither PASS nor your regression verdict, and sort it near the top of the report so a dark gate is still seen.
• Don't let exemption mechanisms such as human-block declarations or skip lists absorb it.
• Keep "found a bad state" on its own code, and keep it loud.
• Test the convention across every gate the runner discovers, not just the one that broke.
• Record the exit code and the stderr tail next to each scheduled verdict. My drift log doesn't, which is why the diagnosis above rests on a cache side effect instead of a log line.

What this does not prove

• The 00:19 diagnosis is inferred from the cache and from what runs the gate. The drift log keeps verdicts only, so the specific cause (DNS, a rate limit, a timeout) is not pinned.
• If the 00:19 failure was an exception the gate doesn't catch, it exited 1 without printing UNMEASURABLE, and the same failure would still read REGRESSION after this fix. Only the errors the gate catches now become UNMEASURABLE.
• The cache argument covers the October 23 run alone. I have no exit codes for the earlier entries in the drift log, and some of those REGRESSION verdicts may have been real reversals.
• The suite is now quieter on measurement gaps, on purpose. A gate that is chronically unmeasurable shows UNMEASURABLE instead of REGRESSION: still visible, but it no longer screams. A gate that should hard-fail when it cannot measure must not use exit 3.
• The guard test reads only the declared EXIT_UNMEASURABLE constant. A gate that used 3 for a real failure without declaring that constant would now be read as unmeasurable, and the test would not notice. A grep of the current gates found none that does, but nothing stops a future one.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.