The triage is the product: running AI agents against Ethereum's protocol code

Notes from the Ethereum Foundation's Protocol Security squad connected moving coordinated AI agents against existent protocol code, including however we signifier the work, what holds up nether scrutiny, and what lawsuit teams and information researchers tin instrumentality from it. This station stands connected its own; aboriginal posts volition spell deeper connected idiosyncratic clients.

What we've been running, and what amazed us

On the Ethereum Foundation's Protocol Security team, we've been moving coordinated AI agents against the kinds of systems the web depends on, similar systems software, cryptographic code, and contracts that person to beryllium right. The agents recovered existent bugs. One is present public: a remotely-triggerable panic successful libp2p's gossipsub, a halfway portion of the peer-to-peer furniture Ethereum statement clients tally on, fixed and disclosed arsenic CVE-2026-34219 with recognition to the team.

Agents uncovering bugs wasn't the surprise. The astonishment was however small of the enactment went into uncovering them, and however overmuch went into telling the existent bugs from the ones that conscionable looked real.

This station is for lawsuit teams and information researchers who privation to bash the aforesaid thing. It covers however we signifier the agents, the barroom a campaigner has to wide earlier it counts arsenic a finding, and the habits that support the results trustworthy.

Teams elsewhere are converging connected the aforesaid recipe. Anthropic's Frontier Red Team built an cause that writes property-based tests and recovered existent bugs crossed the Python ecosystem. Cloudflare ran a frontier exemplary done a security-research harness against their ain systems. Everyone lands connected the aforesaid loop: constituent a susceptible exemplary astatine a codebase, fto it search, and triage what comes back. So the existent question is however to bash this without drowning successful confident-sounding noise.

One caveat up front: tooling for agent-driven audits moves fast, and immoderate circumstantial setup is retired of day successful a fewer weeks. So this station is deliberately astir the methods, which are persistent, alternatively than the tooling. Disclosure is its ain taxable and volition astir apt beryllium its ain post.

An cause pointed astatine a codebase is simply a hunt tool, a batch similar a fuzzer. The quality is what comes back. A fuzzer hands you a clang and a stack trace. An cause hands you a batch more, including a write-up (call chain, interaction claim, suggested severity) and the artifacts to backmost it, similar a proof-of-concept you tin tally against the existent code.

All of that makes the effect casual to work and casual to trust, the moving proof-of-concept astir of all. So don't number however galore candidates an cause produces. Count however galore crook retired to beryllium real.

How the enactment is organized

We tally galore agents successful parallel against 1 target. They coordinate done the repository itself, with shared authorities successful mentation power and nary cardinal process handing retired work. An cause writes down a assertion wherever the others tin spot it, does the work, and commits.

We got this attack from Anthropic's writeup connected building a C compiler with a fleet of agents, which coordinates the aforesaid way. There's nary cardinal coordinator to physique oregon maintain, and little that tin spell wrong.

The roles are generated by the enactment that's discovered:

• Recon turns an onslaught aboveground into concrete, testable hypotheses. Not "audit the decoder" but "this tract is trusted past this point; here's the spot it should keep, the mode it mightiness break, and the impervious that would settee it."
• Hunting takes 1 hypothesis, traces the codification path, and tries to physique a reproducer.
• Gap-filling looks astatine what was accepted and what was rejected, writes the adjacent batch of hypotheses, and tracks sum truthful the agents don't support going implicit the aforesaid ground.
• Validation re-checks each campaigner independently, removes duplicates, and decides.

We didn't invent this pipeline. Cloudflare describes the aforesaid stages, recon, parallel hunting, autarkic validation, deduplication, reporting, and their writeup helped signifier ours.

Here's what a campaigner looks similar earlier it counts arsenic a finding:

The schema is determination for a reason. It forces a specific, testable assertion and a wide explanation of done. An cause that has to constitute down an observable impervious can't autumn backmost connected "this looks risky."

Reproducible oregon it didn't happen

One regularisation matters much than immoderate other. A campaigner isn't a uncovering until there's a self-contained artifact that reproduces the nonaccomplishment against the existent code, and that runs for idiosyncratic who didn't constitute it.

The reproducer doesn't work the write-up, and it doesn't attraction however assured the exemplary sounded. It either runs oregon it doesn't.

Most of its worth is successful the mendacious positives it catches. Three of them travel up implicit and over, and each 1 is the cause getting a walk for the incorrect reason:

• A panic that lone happens successful a debug build. Compile and tally it the mode the bundle really ships, and the worth conscionable wraps around. Nothing crashes. It looks similar a crash, but it isn't one.
• A reproducer that builds immoderate interior worth by hand, 1 nary existent input could ever produce, due to the fact that each way an attacker controls rejects it earlier. The bug lone "reproduces" against a relation that thing reachable calls that way.
• In formal-verification work, a impervious that goes done but doesn't mean what you wanted. The connection is trivially existent careless of what the codification does, oregon it's weaker than the spot you meant to capture. The verifier is satisfied, but the theorem doesn't constrain the behaviour you really cared about.

None of this is new. It's the aforesaid happening arsenic a trial that passes due to the fact that it doesn't really cheque anything. What's caller is the volume. An cause writes the useless mentation arsenic accelerated arsenic the existent one, and conscionable arsenic confidently. So the cheque has to beryllium automatic. You can't number connected the cause to drawback itself.

Signal-to-noise is astir of the work

Most candidates are wrong, duplicate, oregon retired of scope. That's not a occupation with the method; that's however it works. The extremity is to cull the incorrect ones accelerated and backmost the existent ones with impervious that's hard to reason with.

Every campaigner that survives gets 2 autarkic checks. Can a existent attacker really scope it successful a mean configuration? And what does it outgo the attacker to propulsion off, compared to what it costs the web if it works? A bug that immoderate azygous adjacent tin trigger is precise antithetic from 1 that needs peculiar entree oregon a immense magnitude of resources.

Everything gets checked against a moving database of what's already known, fixed, oregon rejected. Without that, the agents support rediscovering the aforesaid closed contented and reporting it again and again.

Acceptance rates alteration a batch from people to target, and that saltation is utile connected its own. Run this against mature, heavy audited codification and astir thing survives, which is inactive worthy knowing. "We looked hard and recovered nothing" is simply a existent result. Run it against less-explored code, oregon against formally verified code, wherever a machine-checked impervious covers a exemplary and the deployed bytecode is lone assumed to lucifer it, and much gets through.

We're not the lone ones who recovered that the triage is the hard part. Cloudflare's main takeaway was that a constrictive scope beats wide scanning. Anthropic's property-based-testing agent generated thing similar a 1000 campaigner reports, past utilized ranking and adept reappraisal to get down to a apical tier that held up astir 86 percent of the time. The procreation was the casual part. I'm not going to people our ain numbers here; tied to a circumstantial target, they'd accidental much astir the people than astir the method.

What the agents are bully at, and wherever they mislead

There's hype successful some directions, truthful here's a plain database of what the agents bash good and wherever they mislead.

The divided isn't adjacent dependable from 1 task to the next. Stanislav Fort, investigating a scope of models connected existent vulnerabilities, calls this a jagged frontier, oregon a exemplary that recovers a afloat exploit concatenation connected 1 codebase tin neglect basal data-flow tracing connected another. You can't presume 1 bully effect means the adjacent volition clasp up, which is different crushed each campaigner gets checked connected its own.

The past enactment is the important one. A azygous cause league is bully astatine one-shot reasoning and atrocious astatine bugs that span a series of steps, wherever each measurement is valid and lone the bid is wrong. For those, the cause isn't the hunt tool. Its occupation is to suggest which sequences are worthy moving done a stateful trial harness. Used that way, it works well. Used arsenic a replacement for the harness, it misses the astir costly bugs determination are, the ones that lone amusement up crossed a sequence.

Keeping it honest

A fewer habits bash astir of the enactment of making cause findings trustworthy, and nary of them are complicated.

• Provenance connected each artifact: what produced it, with what context, against which revision. A uncovering should beryllium thing you tin re-run months later.
• Determinism wherever it counts: 1 environment, 1 mode to physique and run, truthful "reproduces" means the aforesaid happening connected each machine, not conscionable the 1 wherever it was found.
• Norms, not scripts: archer agents what matters, the invariants and the barroom for a existent finding, alternatively of a numbered procedure. Over-scripted agents interruption the aforesaid mode over-specified tests do, they support pursuing the steps aft the steps halt making sense. A study of repository discourse files recovered the aforesaid thing: the other requirements lowered task occurrence and raised outgo by implicit 20%, and the authors urge keeping discourse to the minimal requirements.
• A idiosyncratic makes the last call: agents suggest. They don't determine what's real, what's a duplicate of a known issue, oregon what gets disclosed and when.

The bottleneck moved

AI didn't regenerate the information researcher. It moved the work. The clip that utilized to spell into coming up with and chasing down hypotheses present goes into judging them astatine scale, including gathering the oracle, moving the triage, keeping the database of known issues, and handling disclosure.

The bottleneck didn't spell away. It moved from uncovering bugs to trusting the results, which is simply a amended spot for it, due to the fact that that's wherever quality judgement really matters. But it's inactive a bottleneck, and ignoring that is however you extremity up shipping a incorrect "it's fine."

The practices that marque this enactment aren't new. Reproducible failures, existent oracles, and cautious triage are the aforesaid practices that turned fuzzing from a probe taxable into modular signifier implicit the past 15 years. The tools are new. The practices aren't.

How accelerated the tools support changing is an unfastened question. Nicholas Carlini, cautious and erstwhile a skeptic himself, argues the exponential lawsuit is worthy taking seriously, adjacent portion helium keeps wide mistake bars connected it. If the procreation broadside climbs that fast, the judgement broadside has to ascent with it, oregon the spread betwixt what gets produced and what really gets verified lone widens.

For the systems Ethereum depends on, that's the portion that matters. Agents fto america screen acold much crushed than we could by hand. In exchange, they inquire for much cautious judgment, crossed a overmuch bigger heap of confident-sounding claims. That's a commercialized worthy making, arsenic agelong arsenic you retrieve that the judgement is the existent product.