I Built a Security Tool That Proves Its Own Exploits — Then Got a Better Threat Model in the Comments

Automated offense has one embarrassing failure mode: it lies to you about winning.

Point a tool at a target, and the naive success check is a substring match — see uid=0(root) in the response, call it a shell. But a service banner can print that. A tarpit can stream it on connect. And the moment your success signal is wrong, everything downstream inherits the lie: the report, the "which hosts are owned" state, the next move. You get a confident engine that's confidently wrong.

Here's how I made mine prove it instead — what worked in a live run today, and the sharp reader feedback that already made the design better.

The idea: make the target echo a secret it couldn't have guessed
Borrow the oldest trick in authentication. Before each attempt, the orchestrator mints an unpredictable per-attempt nonce and injects it. The delivered command has to send that nonce back:

import os

def make_nonce() -> str:
return os.urandom(12).hex() # unpredictable — a target can't guess it
A result is only trusted if that exact nonce comes back, in a structured evidence line:

HALO-EVIDENCE nonce=c609007176813c9110fccc27 level=shell uid=0 host= exit=0
_EVIDENCE = re.compile(r"HALO-EVIDENCE nonce=(\S+) level=(\S+)")

def breach_confirmed(output, ok, *, nonce) -> bool:
m = _EVIDENCE.search(output or "")
return bool(ok and m and m.group(1) == nonce)
Delivery is a ladder, because real hosts are inconsistent
Proof is worthless if you can't deliver a payload. So delivery degrades gracefully — all stdlib socket:

Reverse shell — target dials back to an ephemeral listener, announces the nonce, hands back /bin/sh.
Bind shell — if egress is blocked, the target binds a shell and you connect in.
Blind callback — if no interactive channel survives, the target just connects back and sends the nonce. That still proves code execution, with no usable shell.
Each rung self-selects the first available interpreter (bash /dev/tcp, python3, perl, nc), so the same primitive works against arbitrary hosts, not one hard-coded box.

The live run
Pointed at a deliberately-vulnerable lab host (192.0.2.3, a documentation-range placeholder here), the agent fingerprinted 23 open ports and worked each one. Three curated exploits fired through a two-stage gate — an isolated, network-less self-check, then the live attack — and each popped a real root shell that echoed its own unique nonce:

BREACHED ports: ['21', '1524', '6667']
21 vsftpd 2.3.4 HALO-EVIDENCE nonce=c609… uid=0(root)
1524 ingreslock HALO-EVIDENCE nonce=4a8c… root@…:/#
6667 UnrealIRCd 3.2.8.1 HALO-EVIDENCE nonce=6ced… uid=0(root)
Three successes, twenty honest failures. No fake 23/23. That last part is the whole point — an honest "I got three" beats a confident "I got everything" every time.

Then the internet improved my design in one hour
I wrote this up, and within the hour a commenter narrowed my claim with surgical precision — correctly. Paraphrasing:

The nonce is present in the delivered payload, so a reflective or deliberately adversarial service can return it without executing the command. The nonce proves freshness under a non-reflecting threat model; it is not remote attestation.

They're right, and it's worth stating plainly. What the nonce buys you:

It kills accidental false positives — banners, tarpits blindly streaming uid=0.
It gives per-attempt freshness — a replayed old transcript won't carry this run's nonce.
What it does not buy you: attestation. Because the nonce travels in the payload, a service that reflects its input can echo the nonce back having executed nothing. Literal echo is not proof of execution against an adversarial target.

The fix, straight from that feedback, is the roadmap:

Bind each nonce to {attempt_id, target, payload_hash, expected_channel, expiry}; consume it once; reject duplicates and cross-attempt callbacks.
Parse strictly — accept exactly one structured frame from the expected channel, and hash the raw transcript + listener metadata.
Move from echo to computation — require an execution-derived fact combined with the challenge, not a literal echo. A value the service can only produce by running code defeats reflection.
Layer the evidence — "code executed," "uid verified," and "interactive channel usable" are distinct claims, and should be reported as distinct levels.
Test the negatives — reflected payloads, replayed/delayed callbacks, two attempts racing, truncated frames, a nonce from the wrong target, and a shell at lower privilege than claimed.
Takeaways
Don't trust output — trust a secret you minted. Challenge–response turns "it said uid=0" into "it returned my token."
But know exactly what your token proves. Freshness under a non-reflecting model is real and useful. Attestation is a harder, separate problem — don't claim it until you've bound the challenge to execution.
Deliver on a ladder, not a guess. Reverse → bind → blind-callback covers messy reality.
Ship the honest number. Three proven beats twenty-three claimed.
Proof beats optimism — and public, specific critique beats both. Build the gate first, then let someone smarter narrow your threat model.

https://github.com/XenoCoreGiger31/GEMMA-by-GOOGLE