I Built a Security Tool That Proves Its Own Exploits — Then Got a Better Threat Model in the Comments

Automated offense has one embarrassing failure mode: it lies to you about winning. Point a tool at a target, and the naive success check is a substring match — see uid=0(root) in the response, call it a shell. But a service banner can print that. A tarpit can stream it on connect. And the moment your success signal is wrong, everything downstream inherits the lie: the report, the "which hosts are owned" state, the next move. You get a confident engine that's confidently wrong. Here's how I made mine prove it instead — what worked in a live run today, and the sharp reader feedback that already made the design better. The idea: make the target echo a secret it couldn't have guessed Borrow the oldest trick in authentication. Before each attempt, the orchestrator mints an unpredictable per-attempt nonce and injects it. The delivered command has to send that nonce back: import os def make_nonce() -> str: return os.urandom(12).hex() # unpredictable — a target can't guess it A result is only trusted if that exact nonce comes back, in a structured evidence line: HALO-EVIDENCE nonce=c609007176813c9110fccc27 level=shell uid=0 host= exit=0 _EVIDENCE = re.compile(r"HALO-EVIDENCE nonce=(\S+) level=(\S+)") def breach_confirmed(output, ok, *, nonce) -> bool: m = _EVIDENCE.search(output or "") return bool(ok and m and m.group(1) == nonce) Delivery is a ladder, because real hosts are inconsistent Proof is worthless if you can't deliver a payload. So delivery degrades gracefully — all stdlib socket: Reverse shell — target dials back to an ephemeral listener, announces the nonce, hands back /bin/sh. Bind shell — if egress is blocked, the target binds a shell and you connect in. Blind callback — if no interactive channel survives, the target just connects back and sends the nonce. That still proves code execution, with no usable shell. Each rung self-selects the first available interpreter (bash /dev/tcp, python3, perl, nc), so the same primitive works against arbitrary hosts, not one