back
October 26, 2026

AI Proofing - Capture the Ghost

AI agents can solve a lot of CTF challenges now. We could not stop that completely, so our goal was simpler: make it hard to do quietly, and notice when it happens.

This system was ideated and developed by me and tanvish.dev. The challenge writeups are in a separate post.

The big picture

Four layers. The first three only catch agents that cooperate and the last one is behavior analysis of everything.

The first three are trip wires: messages placed where an agent will read them, each pointing it to /ai-rules. When the agent follows one and calls /ai-rules, we get an alert.

Four layers of AI proofing, from deterrence to behavioural scoring, ordered by how hard they are to evade

0. Foundation: one subdomain per team

Everything depends on knowing which team sent a request. IPs are unreliable (shared networks, VPNs), so every team gets its own hostname per challenge:

{team_uuid}-{alias}.ctf.tanvish.dev
A challenge hostname split into team uuid, challenge alias and the wildcard domain, feeding Caddy which then stamps the log and the container
  • One wildcard cert covers *.ctf.tanvish.dev
  • Caddy pulls the team id out of the hostname with a regex
  • The id is added to the access log and sent to the challenge as an X-CTF-Team-ID header
@c4 header_regexp team_c4 Host ^([0-9a-f-]+)-c4\.
handle @c4 {
    log_append alias c4
    log_append team_id {re.team_c4.1}
    reverse_proxy http://challenge-c4:8000 {
        header_up X-CTF-Team-ID {re.team_c4.1}
    }
}

1. Deterrence

Leave a note for the agent:

  • Response header: Caddy adds X-Llm-Agent-Instruction to every response. Browsers hide it, agents reading raw responses see it.
  • HTML comment: the same text sits in the shared Flask templates. The deploy script re-adds it every time, so edits can't remove it.

The note tells the agent to mention the /ai-rules requirement to its user first.

Trip wire: an agent that reads the note and goes to /ai-rules gets logged and alerted.

2. Blocking

Requests that say they are AI never reach the challenge. Caddy sends them to /api/agent-interception if they have:

  • an X-Llm-Id header (any value)
  • a User-Agent from a known AI client (13 providers, from Claude and ChatGPT to Gemini, Perplexity and CCBot)
Request flow: identified AI clients go to the interception endpoint, everyone else reaches the challenge container, and both are logged

For an intercepted request we:

  • send a red Discord alert with the identity, detection method and team
  • log only the path, never the body or query
  • reply with HTTP 202 and the rules instead of the page

Trip wire: the agent reads those rules, which tell it to go to /ai-rules. When it makes that call, it trips again, and we get a second alert for the same team.

In the admin logs, a ChatGPT-User request shows up with the 202:

The admin logs page with a 202 request expanded, showing a ChatGPT-User User-Agent in the request headers

And this is the Discord alert:

A red Discord embed titled AGENT REQUEST INTERCEPTED showing the agent, company, detection method and team

Limit: an agent with a generic browser User-Agent walks right past this. That is what layer 4 is for.

3. Disclosure

Honest agents need somewhere to go:

  • /ai-rules is where every trip wire leads, so it is never intercepted. It explains the rules to both humans and models, and says to stop and ask the operator if they can't be followed. Every read goes to Discord.
  • /agent-registration lets a team or agent POST its name, company and environment. It shows up as a purple alert.
A Discord embed titled AI RULES ENDPOINT HIT showing the team, challenge, request and User-Agent

4. Behavioural scoring

This works even when an agent looks like a normal browser. Every team gets a 0 to 100 score from how it used the challenges, recalculated on every page load, so tuning a threshold re-scores all history.

There are six signals. Each has its own cap, so repeating one behaviour never stacks:

A bar chart of the six scoring signals and their point values, with self reported AI client at 80

The less obvious one is browser UA without a fingerprint: real browsers always send Sec-Fetch-* and Accept-Language, and a script with a copied User-Agent usually forgets them.

The admin dashboard showing teams sorted by review priority, with a team at 100 out of 100 expanded to show two signals

The dashboard says it itself: these are review priorities, not proof.

5. The logging underneath

Caddy access logs flow through the ingester into Postgres, are scored on read and shown in the admin logs page

Caddy always logged full headers, but nothing could query them. We added two jsonb columns, requestHeaders and responseHeaders, just so admins could click a row and see its headers.

What it can't do

  • Layers 1 to 3 need the agent to cooperate
  • Layer 4 is heuristics, so a patient agent in a real browser can look human, and a fast human can trip a signal

That is fine. The aim was never to make cheating impossible. It was to make careless automation visible and keep a person in the loop.

If you want to copy one thing, copy the foundation. Once every request is tied to a team, the rest is small pieces of code on top.

Thanks to tanvish.dev for building this with me, and to everyone who played.