My test suite had 100% coverage. Mutation testing still found real bugs.

breakwater is my resilience toolkit for Node.js — retry, circuit breaker, timeout, bulkhead, rate limiting, all composable, with observability built in. It's the library that stands between your service and a dying dependency, which means its tests are not decoration. If the circuit breaker's threshold logic is wrong, somebody's incident gets worse.

The suite had 100% line coverage on the core. Every branch tool was green. I was feeling pretty good about it.

Then I ran Stryker and it graded my suite at 80.52%.

Coverage measures execution. Mutation measures assertion.

If you haven't used mutation testing: the tool takes your source, makes one small change at a time — flips a >= to >, replaces a condition with true, empties a block — and runs your suite against each mutant. If some test fails, the mutant is killed. If everything stays green, it survived: your tests executed that line and asserted nothing about it.

Coverage tells you the line ran. Mutation testing tells you whether anyone would notice if the line were wrong. Those are very different claims, and the gap between them is exactly where production bugs live.

Stryker generated 1,114 mutants from my source. 199 survived.

What the survivors actually were

I went through every single one. They fell into five buckets, and each bucket taught me something different.

1. A real bug, in production, that day

A cluster of survivors pointed at the code that injects a registry name into policy options. Writing the test that should have killed them uncovered an actual defect: a named policy entry with an explicit display name reported the pipeline under one name and its rate limiter under another — the same policy showing up as two series in a metrics dashboard, in direct contradiction of the documented "explicit names always win" rule.

That fix shipped to npm the same week. No user had reported it. Coverage had been green the whole time.

2. A test that was lying about what it tested

One test existed specifically to pin a floating-point rounding fix: retryAfterMs must always be sufficient — wait exactly that long and you're admitted. Its comment proudly said the configuration reproduced a case where the naive ceil() lands one millisecond short.

Except it didn't anymore. The configuration (interval: 8737) no longer triggered the correction loop at all — the test passed with or without the code it claimed to protect. I searched the parameter space for a configuration that actually hits the rounding error (interval: 161, if you're curious: 161 * (1/161) is 0.999…) and rewrote the test around it.

A test that asserts something true but unrelated to its purpose is worse than no test: it's documentation that lies.

3. Documented contracts nobody ever asserted

The biggest bucket. Things the README and docs promised, that all worked, and that no test would defend against regression:

• The circuit breaker's open, close and halfOpen events — only the generic stateChange was ever checked. Payloads, correlation IDs, one-event-per-transition: all unpinned.
• Every distributed state-store path. The breaker has a pluggable StateStore designed for sharing circuit state across instances — stores that answer asynchronously, losing a compare-and-set race to a peer, another instance winning the half-open probe election. The interface existed, the local implementation was tested, and not one distributed behavior had a test.
• Boundary semantics: a failure rate landing exactly on the threshold, a retry delay landing exactly on the deadline.
• Jitter that actually spreads. My tests asserted delays stayed within [0, max] — a jitter implementation returning a constant would have passed while doing nothing about thundering herds.
• A metrics collector implementing none of its optional callbacks. This one hid behind deliberate error-swallowing (monitoring must never break execution), so the fix asserts the error reporter stays silent too.

4. Dead code — the report doubles as a detector

Some survivors survived because no input can reach them. Two examples from the timeout policy: a guard in the timer callback protecting an assignment that nothing could ever read (the catch block re-checks the same condition first, and signals never un-abort), and one arm of a condition that was provably always false at its only call site.

I didn't take "provably" on faith — I rebuilt the old version and ran a differential matrix across cooperative/aggressive modes, abort orderings and rejection identities. Zero divergence. Both paths deleted. The invariant they pretended to guard now lives in exactly one place, with a test pinning its sharpest corner: an external cancellation that lands after the deadline fired still wins, and the timeout event stays silent.

5. Equivalent mutants — know when to stop

Not everything should be killed. { once: true } on an abort listener is unkillable by any black-box test, because an AbortSignal fires abort at most once in its lifetime — the mutant is behaviorally identical. Same for a fast-path that skips building a composite signal: pure allocation optimization.

Chasing those means asserting listener counts and internal allocations — testing implementation, which is the same disease as coverage-theater in the opposite direction. I documented each accepted survivor and moved on. The honest end state isn't 100%: it's every survivor analyzed and either killed or justified.

Final score: 95.11%, with the suite growing from 157 to 223 tests. And one bonus find that wasn't a mutant at all: two of my new tests would have hung CI forever instead of failing if their behavior regressed, because node:test has no default timeout. If you use the built-in runner, set one:

"test": "node --import tsx --test --test-timeout=10000 'tests/**/*.test.ts'"

The setup, if you want this on node:test

Stryker's tap runner drives the native runner just fine, TypeScript included:

{
  "testRunner": "tap",
  "tap": {
    "testFiles": ["tests/**/*.test.ts"],
    "nodeArgs": ["--import", "tsx", "--test", "--test-reporter=tap", "--experimental-test-isolation=none"]
  },
  "mutate": ["src/**/*.ts"],
  "coverageAnalysis": "perTest",
  "reporters": ["json", "html", "progress"],
  "timeoutMS": 20000
}

coverageAnalysis: "perTest" matters — it runs only the tests covering each mutant, which took a full run of my suite to under six minutes. I keep it as a manual gate (npm run test:mutation) rather than a CI job: it's a code-review tool, not a merge blocker.

Why I did all this before shipping observability

Because the newest release of breakwater is an observability feature, and shipping a tool that watches production on top of tests I couldn't trust felt absurd.

breakwater@0.7.0 adds breakwater/prometheus — ready-made prom-client collectors for every signal the library emits:

import { resilience } from 'breakwater'
import { prometheusCollector } from 'breakwater/prometheus'

const payments = resilience({
  name: 'payments-api',
  retry: { attempts: 3 },
  circuitBreaker: { consecutiveFailures: 5 },
  timeout: 2_000,
  metrics: prometheusCollector()
})

That's the whole integration. Eight metrics come out: executions and a duration histogram by outcome, retries, timeouts, fallbacks, rejections by reason (circuit_open, bulkhead_full, rate_limited…), the circuit state as an enum gauge, and a transitions counter. prom-client is an optional peer dependency — the core keeps its zero runtime dependencies, and importing plain breakwater never loads it.

Two design notes that came straight out of caring about honest observability:

Labels are low-cardinality by construction. Correlation IDs and attempt numbers never become label values. The only free-form label is your policy's name.

Absence means healthy. The state gauge is event-driven, so a circuit that has never tripped exports no state series yet. The docs say so explicitly, and the suggested alert targets the gauge (breakwater_circuit_state{state="open"} == 1) instead of increase() on a transitions counter — a counter series born by its first-ever transition needs two samples before increase() sees anything, which is exactly how you miss your first incident.

There's a ready-to-import Grafana dashboard covering all eight metrics, and a runnable demo — docker compose up brings up a breakwater-protected app with a scripted outage every 90 seconds, Prometheus scraping it, and Grafana already provisioned. You get to watch the circuit open, fallbacks take over, and a half-open probe close the loop, on a real dashboard, without wiring anything.

And yes: the adapter's module was born under the mutation gate. First report: 100% — zero survivors. It turns out writing honest tests is much easier when you do it from the start.

Takeaways

• Coverage is a floor, not a verdict. 100% coverage coexisted with 199 unasserted behaviors in my suite.
• Read every survivor. The value isn't the score — it's the taxonomy: real bugs, lying tests, unpinned contracts, dead code, and equivalents you consciously accept.
• Mutation reports find dead code that no linter flags, because reachability is a semantic property.
• Set a test timeout with node:test. A test that hangs instead of failing is a CI outage waiting for a regression.
• Don't chase 100%. Killing equivalent mutants means testing internals. Analyze, justify, stop.

breakwater is on GitHub and npm — MIT, docs for every policy, and the comparison with opossum/cockatiel is in the README. If you run it behind Prometheus, I'd genuinely like to hear what your dashboards catch.