Verify agents by signed requests, not by IP or User-Agent

Decide how to treat automated traffic from a cryptographic signature you can verify, not from a User-Agent string or an IP range, so an agent acting for a real user is not blocked as a scraper and a scraper cannot pass as that agent.

Why

Most sites tell bots from people with heuristics: User-Agent strings, IP reputation, CAPTCHAs, rate patterns. Mathias Biilmann describes what that does to agents: all the heuristics and roadblocks built for bots tend to impact agents equally, and he learned it when Google disabled one of his accounts after detecting the personal agent he had connected to his own Gmail and calendar. He calls for real standards that let agents identify themselves as legitimate actors operating on someone's behalf. The IETF chartered the Web Bot Auth working group for that problem. Its charter notes that origins may want different service levels for automated and human traffic, and that the current tools (IP allowlisting, User-Agent strings, shared API keys) have significant limitations in security, scalability and manageability. A User-Agent can be copied by anyone, and an IP range says nothing once traffic passes through shared infrastructure. The building block is RFC 9421, HTTP Message Signatures, a Standards Track way to sign components of an HTTP request so a verifier can check them even after intermediaries. Web Bot Auth applies it to bots: the operator publishes its public keys at /.well-known/http-message-signatures-directory and signs each request; Cloudflare already verifies such signatures for verified bots and agents. The protocol is still an IETF draft (draft-ietf-webbotauth-httpsig-protocol-00, September 2026), so verify signatures where your edge supports them and keep identity separate from permission: a verified signature tells you who is calling, and your access rules still decide what they may do.

Do

Accept and verify signed agent requests, publish which signed agents you allow, and base access on verified identity plus behavior.

Don't

Allowlist or block agents by User-Agent string or IP range alone.

Artifact

GET /products/42 HTTP/1.1
Host: shop.example.com
Signature-Agent: "https://signature-agent.test"
Signature-Input: sig2=("@authority" "signature-agent");created=1735689600;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";alg="ed25519";expires=1735693200;tag="web-bot-auth"
Signature: sig2=:jdq0SqOwHdyHr9+r5jw3iYZH6aNGKijYp/EstF4RQTQdi5N5YYKrD+mCT1HA1nZDsi6nJKuHxUi/5Syp3rLWBA==:

The verifier fetches the public key from https://signature-agent.test/.well-known/http-message-signatures-directory, checks the signature and the expires time, and only then applies its rules for that agent.