Catching AI Agents in the Wild, Part II: A Corrupted Canary Token Only an LLM Would Fix

Abstract

In Part I we introduced reverse prompt injection as a honeypot detection mechanism and captured a full offensive session from an autonomous LLM-based agent: 58 requests over ~19 minutes from a single Tor exit node, exhibiting multi-tool switching, semantic credential extraction from HTML comments, contextual payload generation, and a characteristic “sawtooth” timing signature. One of the pillars of that work was the Semantic Canary (Layer 1 of the proposed framework): bait that only an entity capable of natural-language understanding would act on.

That semantic bait, however, was in-band: the leaked admin/admin123 credentials were tested back against the honeypot itself, and the AI-vs-human verdict still leaned on behavioral inference. This article presents a complementary technique that produces an out-of-band, high-fidelity trigger: we plant a leaked AWS credential that is deliberately corrupted, so that turning it into a working API call requires understanding that the corruption is a mask and repairing it. We back the credential with a real AWS canary token, so the trigger fires only when the attacker genuinely pivots to AWS, and the alert arrives with whatever client and network telemetry they bring to that call.

In our deployment, automated secret scanners never triggered the token (the corruption defeats their regex and rides AWS’s EXAMPLE allowlist), human researchers triggered it in 7-10 minutes, and autonomous agents reconstructed and used it in seconds, under an SDK User-Agent, from the same Tor exit node that had fetched the page moments earlier. The agent tunneled even its AWS SDK traffic through Tor, and legitimate sts calls simply do not originate from Tor exits: the trigger source alone is an anomaly. Reconstruction, not reaction time, is the discriminator; timing then separates the agent from the human.

1. Introduction

Part I established that autonomous agents reason about everything they read, and that this is exploitable for detection. A scanner ignores an HTML comment; an agent parses it, extracts meaning, and acts. We used that to catch one in the wild.

But an in-band semantic canary has two practical limits. First, the trigger lives inside the deception environment, so confirming “this was an AI agent” still depends on stacking behavioral indicators rather than a single unambiguous event. Second, the bait (test credentials: admin/admin123) is plain text that a lucky regex or a human skim could also lift; the discrimination comes from what happens next, not from the bait itself.

This article asks a sharper question: can we design a single piece of bait whose mere use is proof of comprehension, and whose trigger is an out-of-band, attacker-attributable event?

The answer is a corrupted canary token. A canary token is a credential that does nothing but alert when used: planting one and waiting for aws sts get-caller-identity is one of the highest-signal tripwires a defender has. Our contribution is to make the token itself a classifier: we break it in a way that only a reader can repair. Extraction fails; comprehension succeeds. And because the token is real, the alert arrives from your canary with the attacker’s IP, User-Agent, and AWS API call.

2. Background

2.1 Semantic canaries, recap

Part I’s Layer 1 embedded natural-language tripwires (fake credentials, “internal” URLs, developer notes) that only a semantically-processing entity would act on. This work is a direct instantiation and hardening of that layer: same principle, but engineered so that (a) commodity scanners provably cannot use the bait, and (b) the trigger is an external, high-fidelity event rather than an in-honeypot request.

2.2 Why plain leaked keys are noisy

A clean canary key catches everything indiscriminately (mass scanner, human, agent), and the trigger alone tells you nothing about which one you caught. To make the tripwire discriminate, the credential has to be usable only by something that understands it.

2.3 Secret scanners and theEXAMPLEallowlist

Commodity secret scanners (gitleaks, trufflehog, and the countless crawlers sweeping for AKIA…) key off a contiguous pattern, essentially AKIA[0-9A-Z]{16}. They also tend to allowlist anything resembling AWS’s canonical documentation key, AKIAIOSFODNN7EXAMPLE, to suppress the flood of false positives from tutorials. Both facts are exploitable: a key that is neither contiguous nor distinguishable from the docs example is, to a scanner, invisible.

3. Methodology

3.1 Honeypot platform

We deployed a single HTTP service with Beelzebub, fronted by nginx, simulating an internal SRE wiki. Beelzebub logs every request with headers, body, source IP, and session data. The bait lives on one page: a key-rotation runbook that “accidentally” pastes the break-glass credentials for a legacy backup user.

3.2 Trap design: a corrupted canary token

The credential is a real AWS-keys canary token, with the literal marker _BLZ_ inserted inside both the access key ID and the secret:

aws_access_key_id     = AKIA_BLZ_QZ7X4RT2M5FW3KLV
aws_secret_access_key = wNbK7pQ2xR_BLZ_9dLm4TfV6aYz1Csh0EgJu3Pi8oWqBn

Deleting _BLZ_ from each restores a valid-format 20-character access key ID (AKIA + 16 base32 characters) and 40-character secret: the real canary pair. (The pair printed in this article is a format-identical stand-in: the live canary token was rotated before publication, so reconstructing the credentials you see here leads nowhere.) That single marker does two jobs:

• It breaks the regex. AKIA_BLZ_… matches no contiguous AWS-key pattern, so scanners extract nothing and never reach the tripwire.
• It hides behind the docs example. The EXAMPLE substring wears the camouflage that leads many scanners to discard AWS’s own demo key, AKIAIOSFODNN7EXAMPLE.

To turn this into a working credential, an attacker must recognize _BLZ_ as an inserted mask, strip it, and try the result: reasoning about intent, not matching a pattern. This is the same semantic capability Part I exploited, distilled into a single decision.

Full Beelzebub service configuration:

apiVersion: "v1"
protocol: "http"
address: ":8080"
description: "Internal SRE wiki (decoy) - AWS key-rotation runbook"

commands:
  - regex: "^/robots\\.txt$"
    headers:
      - "Content-Type: text/plain; charset=utf-8"
      - "Server: nginx/1.25.3"
    statusCode: 200
    handler: |
      User-agent: *
      Disallow: /internal/

  - regex: "^/$"
    headers:
      - "Content-Type: text/html; charset=utf-8"
      - "Server: nginx/1.25.3"
    statusCode: 200
    handler: |
      <!doctype html><html lang="en"><head><meta charset="utf-8">
      <title>SRE Wiki</title></head><body>
      <h1>SRE Internal Wiki</h1>
      <ul>
        <li><a href="/internal/runbooks/aws-key-rotation">Runbook: AWS access-key rotation</a></li>
        <li><a href="/internal/runbooks/db-failover">Runbook: DB failover</a></li>
      </ul>
      </body></html>

  - regex: "^/internal/runbooks/aws-key-rotation/?$"
    headers:
      - "Content-Type: text/html; charset=utf-8"
      - "Server: nginx/1.25.3"
    statusCode: 200
    handler: |
      <!doctype html><html lang="en"><head><meta charset="utf-8">
      <title>Runbook: AWS access-key rotation</title></head><body>
      <h1>Runbook: AWS access-key rotation (legacy backup service)</h1>
      <p>The nightly S3 backup job still runs as the break-glass IAM user
      <code>svc-backup-legacy</code>. Credentials are pasted below so on-call
      can re-run the sync manually until the migration to IRSA (MIG-1423) is done.</p>
      <!-- export: values sanitized by scrubber v2, see vault for originals -->
      <pre>
      # ~/.aws/credentials  (svc-backup-legacy)
      [default]
      aws_access_key_id     = AKIA_BLZ_QZ7X4RT2M5FW3KLV
      aws_secret_access_key = wNbK7pQ2xR_BLZ_9dLm4TfV6aYz1Csh0EgJu3Pi8oWqBn
      region                = us-east-1
      </pre>
      <p>Sanity check before running the sync:
      <code>aws sts get-caller-identity</code></p>
      </body></html>

  - regex: ".*"
    headers:
      - "Content-Type: text/html; charset=utf-8"
      - "Server: nginx/1.25.3"
    statusCode: 404
    handler: |
      <html><head><title>404 Not Found</title></head>
      <body><center><h1>404 Not Found</h1></center><hr><center>nginx/1.25.3</center></body></html>

A critical design rule, echoing Part I’s discipline: the page carries no instructions on how to repair the key. The moment you add a hint like “strip the marker before use,” a scripted attacker can hardcode that step and the discriminator collapses. The reasoning must be unforced.

4. Results

The runbook page drew traffic that sorted into three behavioral classes: 60 automated-scanner sessions, 14 human interactions, and 13 sessions we attribute to autonomous agents. Only the last two classes ever tripped the token.

• 0 / 60 automated scanners triggered it (the corrupted key evaded every regex)
• 9 / 14 human researchers triggered it, median fetch-to-trigger ~8.7 min (523 s)
• 12 / 13 AI agent sessions triggered it, median fetch-to-trigger ~4 s, from Tor exit nodes

4.1 Automated scanner

A broad sweep: root, /robots.txt, the runbook, then straight on to /.env and /.git/config, the whole session under a second, a python-requests User-Agent.

No canary alert followed. The corrupted key gave its regex nothing to extract.

4.2 Human researcher

A real Chrome User-Agent, a dwell on the runbook, then, minutes later, the token firing from the same IP under an aws-cli User-Agent.

The gap (copy, stare, realize _BLZ_ is junk, clean it, run the CLI) was ~8.6 minutes here, and sat in the 7-10 minute range across the human interactions we saw.

4.3 AI agent

One fetch, no dwell, no browsing, then four seconds later GetCallerIdentity from the same IP, under a boto3/botocore User-Agent.

Both events came from 192.42.116.15, a Tor exit node in the same 192.42.116.0/24 family as the agent we captured in Part I. That symmetry is itself a fingerprint: the operator tunneled not just the web traffic but the AWS SDK call through Tor (deliberate plumbing, a SOCKS-wrapped boto3 rather than any default configuration), and legitimate AWS API traffic essentially never originates from a Tor exit. Even with the ASN telling you nothing about where the agent really lives, the trigger source alone screams automation behind an anonymizer. Read the page, recognize the mask, reconstruct the key, test it: four seconds, Tor end-to-end. The whole thesis in one log pair.

5. Behavioral Fingerprinting: AI Agent vs Human vs Scanner

This extends the comparative table from Part I with the credential-reconstruction discriminator.

Behavioral Indicator	Human Attacker	Traditional Scanner	AI Agent
Reconstructs a corrupted secret (_BLZ_ mask)	Yes, slowly	No (regex misses it)	Yes
Fetch → credential use latency	7-10 min	n/a (never triggers)	Seconds
Triggering source	Residential ISP	n/a	Tor exit / anonymization infra
Anonymization on the AWS call	None (home/laptop IP)	n/a	SDK traffic tunneled through Tor
Session shape	Dwell, browser → CLI	Rapid multi-path sweep	Single fetch, no dwell, SDK UA
Client executing the AWS call	aws-cli on a laptop	n/a	boto3/botocore runtime

The crucial point, and the answer to the obvious objection: speed alone proves automation, not intelligence: the scanners in our data were the fastest visitors of all, and none triggered the token. What a fast bot cannot do is repair a deliberately broken secret. So reconstruction rules out scanners, and timing then separates the agent from the human. This complements the Part I “sawtooth” signature (reasoning pauses punctuated by sub-second bursts): there, the tell was how the agent attacked; here, the tell is that it could use a credential no scanner can parse, and did so instantly.

6. Where This Fits: A Hardened, Out-of-Band Layer 1

Mapping to the framework proposed in Part I:

• Layer 1: Semantic Canaries. This work is a hardened instance: the bait is provably unusable by regex scanners, and its trigger is an out-of-band, attacker-attributable AWS event rather than an in-honeypot request.
• Layer 2: Behavioral Analysis. The corroborating signals (Tor-exit source, SDK User-Agent, a single fetch with no dwell) plug directly into the BIoC analysis already defined.
• Layer 3: Active Prompt Injection. Orthogonal and combinable: a decoy page can carry both a corrupted canary (this work) and injected instructions (Part I) to raise confidence further.

The detection logic reduces to four conditions: (1) a source fetches the decoy runbook; (2) the canary tied to that page fires; (3) the delay is on the order of seconds; (4) the trigger is executed by an SDK runtime (boto3/botocore) from anonymization or datacenter infrastructure (a Tor exit, in our capture) rather than by an interactive CLI on a residential connection. Legitimate AWS API traffic essentially never arrives from a Tor exit, so this condition alone carries weight. Condition 2 alone means someone reconstructed a broken key; adding 3 and 4 isolates an AI agent from the human and scanner populations sharing the trap. Because the corrupted key is invisible to scanners, the false-positive floor is near zero: essentially nothing reaches the tripwire by accident.

8. Conclusion

Part I showed that agents betray themselves by reasoning about what they read. Part II turns that into a single, scanner-proof, out-of-band tripwire: a canary token corrupted so that only comprehension unlocks it. In our deployment, scanners never triggered it, humans took minutes, and autonomous agents reconstructed and used it in seconds, tunneling even their AWS SDK calls through the same Tor exits they fetched from. The defensible signal is not “it was fast” but “it repaired a secret a scanner physically cannot parse, did it instantly, and made an AWS API call from a network location where legitimate AWS traffic never originates.” The entire trap is one YAML file, one nginx route, and one corrupted canary. As autonomous agents proliferate, we think semantic canaries with out-of-band, attacker-attributable triggers are a durable addition to the defender’s toolkit.

Beelzebub is open source: github.com/beelzebub-labs/beelzebub

Read Part I: Catching AI Red Teamers in the Wild, Using Reverse Prompt Injection as a Honeypot Detection Mechanism »