Your leak test is probably measuring the test runner

I have a test that checks a native buffer gets freed. Yesterday I ran it two
ways, twenty seconds apart, on the same machine against the same code.

Alone, it measured 5.11 MB of growth.

As part of the full suite, it measured 489.13 MB.

Nothing about the code under test changed. The only difference was what else
was running at the same time.

Why this happens

ProcessInfo.currentRss returns the resident set size of the process. For
a package that allocates through FFI that is exactly what you want, because
native allocations never show up in Dart heap statistics. It is also the
problem: the number includes everything else the process is doing.

And dart test runs your suite files concurrently, in one process, by default.

So a leak test that reads currentRss is reading a counter that another test
file is also writing to. When my JSON parsing suite ran next to it, the number
moved by nearly half a gigabyte. The bound that decides whether that test passes
is 16.4 MB. The noise is thirty times the entire decision.

The failure mode is nastier than a wrong number. The test passes when you run
it by itself, which is what you do while writing it, and fails in CI where the
whole suite runs. That reads like flakiness. You retry the job, it fails again,
you widen the threshold until it stops complaining, and now the test cannot
detect anything.

Move the measurement into a child process

Move the allocation loop into its own script and run it with
Process.runSync. The child sees only its own work.

// test/leak_probe.dart
void main(List<String> args) {
  final mode = args[0];
  final iterations = int.parse(args[1]);

  // ... build `cycle` for the mode under test ...

  for (var i = 0; i < warmup; i++) {
    cycle();
  }
  final before = ProcessInfo.currentRss;
  for (var i = 0; i < iterations; i++) {
    cycle();
  }
  final grown = (ProcessInfo.currentRss - before) / 1048576;
  print('RSS_DELTA_MB=${grown.toStringAsFixed(2)}');
}

And from the test:

final result = Process.runSync(Platform.resolvedExecutable, [
  'run',
  'test/leak_probe.dart',
  mode,
  '$iterations',
]);

final reported =
    RegExp(r'RSS_DELTA_MB=(-?[0-9.]+)').firstMatch(result.stdout as String);
expect(reported, isNotNull,
    reason: 'leak_probe printed no measurement: ${result.stdout}');
final grownMb = double.parse(reported!.group(1)!);

Two details in there are not decoration.

The warm-up loop. The first iteration pays for library loading, runtime
dispatch and the allocator's initial arena. Without it you attribute all of
that startup to whichever thing you measured first. I once benchmarked two APIs
and got 231 ms versus 2 ms, then added warm-up and got 1.3 ms versus 2.5 ms.
The ordering alone produced a hundredfold error.

The marker. The child prints RSS_DELTA_MB= and the test greps for it
rather than parsing the last line. dart run writes build-hook progress to
stdout, so the output is not just your number. Parsing stdout.trim() works
until the day a hook logs something, and then it breaks in a way that looks
like your code changed.

Here is the same measurement taken all three ways, this morning:

Derive the threshold, do not pick it

A separate process gives you a clean number. It does not tell you what to
compare it against, and this is where I had been guessing.

My original bound was 50 MB. I picked it because it was round and comfortably
above what I saw when the test passed. The payload it was parsing is 4 KB.

A threshold chosen that way is not an assertion. It is a wish with a number
next to it, and it will accept anything that fits under the wish.

Derive it from the payload instead. Ask what the smallest leak on this code
path would actually cost over the iteration count, then bound well under that:

for (final probe in const [
  ('parseBytes',  'padded input and the tape behind at()',      65.5),
  ('decodeBytes', 'the tape and the native copy of the input',  65.5),
  ('ndjson',      'the tape and the native copy of the input',  92.1),
  ('pointer',     'the native copy of a long JSON pointer',     46.9),
]) {
  final (mode, what, smallestLeakMb) = probe;

  test('$mode releases $what', () {
    // ... run the probe, parse grownMb ...
    expect(grownMb, lessThan(smallestLeakMb / 4),
        reason: 'grew ${grownMb}MB over $iterations iterations; losing the '
            'smallest native buffer on this path would cost ${smallestLeakMb}MB');
  }, testOn: '!windows');
}

Each number is the cost of dropping the single cheapest free on that path.
The bound sits at a quarter of it. That leaves room for allocator behaviour
while staying far below any real leak, and it means the four paths have four
different bounds, because they allocate different things.

One more thing about the source data: use bytes that do not compress. I had a
test feeding a flat grey PNG through an image pipeline, and it compressed so
well that the buffers I was trying to observe never got big enough to see.

Verify the test can fail

All of the above is still just a story about a number until you check the one
thing that matters: delete a free and confirm the test turns red.

Do it per-call, not all at once. When I first ran this audit across my
packages, the leak test in one of them stayed green with all four native
frees removed, because the total still landed under the guessed threshold. Any
single one of them, deleted on its own, has to fail. If it does not, that call
site is unprotected regardless of what the suite says.

While doing exactly this, my mutation script reported everything green on the
first run. The script was fine. The Python assert that applied each mutation
was failing silently, so nothing was ever mutated. A check passing does not
mean it ran.

What RSS cannot answer

I deleted a test during this work and did not replace it.

It asserted that a NativeFinalizer reclaimed a document nobody closed, by
measuring resident set after a fixed amount of work. That is a bet on when the
garbage collector runs. It passed locally and failed on Linux CI. A rewrite
comparing dropped references against retained ones was no better: measured back
to back, the two arms came out 123 MB and 177 MB, so the margin it claimed was
not there.

RSS does not fall when native memory is freed, because the allocator keeps the
pages. It cannot answer "was this reclaimed," and the collector makes no timing
promise a test could hold it to.

Deleting that test made the suite honest. What replaced it is coverage of the
things that are actually deterministic: using a closed handle throws, closing
twice is safe, malformed input raises rather than corrupting memory.

Checklist

• Measurement runs in a child process
• The child warms up before the timed loop
• Output uses a marker the parent greps for, not the last line
• Each allocation site is measured in its own loop
• The threshold is derived from the payload, not picked
• Source data does not compress
• Deleting any single free turns the test red

The last one is the cheapest and the only one that proves the rest.

Numbers here were measured on Dart 3.11.0, macOS 26.6, Apple M4, against
simdjson_dart. Written with AI assistance for research, running the
measurements and editing; the findings and the code are from my own packages.