terms.txt signs a crawler's stated purpose. It cannot prove what happens next
robots.txt can ask a named crawler to stay away from a path. It cannot verify who sent the request, distinguish search indexing from model training, state a price, carry a subscriber's permission, or produce a receipt. Nobody designed the file to negotiate with a live AI agent acting for a person.
An independent proposal called terms.txt tries to fill that gap at the website's origin. It combines a public policy file with cryptographically signed requests, signed intent, optional delegation and payment tokens, HTTP 402 responses, and signed delivery receipts. Its reference implementation added 0.20 to 0.65 milliseconds per request in one author's loopback benchmark on a shared CPU core.
The proposal avoids claiming that signatures solve consent. Instead, it separates what a server can enforce before sending a page, what it can audit afterward, and what still depends on contracts or trust. A crawler can sign a false purpose just as easily as a true one.
Why robots.txt cannot carry this negotiation
The Robots Exclusion Protocol was standardized as RFC 9309 in 2022 after decades of use. It maps user-agent names and path patterns to allow or disallow rules. The RFC explicitly says those rules are not access authorization.
Many current policy choices do not fit inside that file. A publisher may welcome search indexing, allow academic archiving, charge commercial training crawlers, and let a subscriber's personal agent read premium pages. One crawler may also serve several purposes. A user-agent string proves none of them.
The terms.txt paper uses recent traffic measurements to explain why this distinction has become expensive. Cloudflare Radar's AI Insights tracks crawler purpose and crawl-to-refer ratios, but the paper warns that these measurements have different denominators, missing referrals, and a commercial conflict because Cloudflare also sells crawler controls. The data motivates a protocol; it does not settle the value of any individual crawl.
The request arrives with identity and intent
A site publishes /.well-known/terms.txt. Each path block can set an unsigned policy and separate rules for search, agent access, training, research, or archiving. A rule can allow, deny, or charge, then add a use ceiling or delegation scope.
Path: /articles/
Unsigned: allow
Purpose: search allow use=reference
Purpose: agent allow use=reference delegation=read:articles
Purpose: train-ai charge 0.002 USD/request
Purpose: research allow use=full
The file alone remains a declaration. Enforcement starts when the automated client signs an HTTP request. The proposal builds on the IETF's active Web Bot Auth draft, which uses HTTP Message Signatures and a key directory at the operator's origin. Signature-Agent names that operator. An Access-Intent header states purpose and intended use. A user-delegated agent can add a scoped, agent-bound token, while a charged request can add a payment voucher.
The origin checks the signature, replay nonce, intent, path policy, and any required delegation or payment. It can return 401 for a failed identity check, 403 for a refusal, 402 with price and terms, or the content plus an Access-Receipt. The origin signs that receipt and appends it to a hash-chained log.
The IETF's AI Preferences work is related but intentionally narrower. It is developing vocabulary for uses such as AI training and search, plus ways to attach preferences to HTTP responses and robots.txt. Its charter excludes authentication and enforcement. terms.txt borrows that vocabulary, then adds a request-side exchange.
A receipt records delivery, not downstream use
The protocol can prove that a request came from the holder of an operator key and that nobody altered its signed purpose in transit. It can prove that the origin delivered a path under a particular version of its terms. It can also stop unsigned requests from protected paths or require payment before delivery.
It cannot prove the declared purpose was honest. Once the bytes leave the origin, HTTP cannot show whether they entered a training set, appeared in full inside an answer, or were retained longer than promised. Receipts make later behavior attributable and give sites evidence for revocation or a contract dispute. They do not make content-use terms self-executing.
Content Credentials can verify an image's history, but cannot tell you it is true describes a similar limit. Cryptographic provenance answers a bounded question. Calling it a truth detector would create false assurance.
The prototype is cheap in latency, costly in throughput
The terms.txt reference implementation is about 600 lines of dependency-free JavaScript for Node.js 22. Its harness runs 24 correctness checks. The published benchmark used one 2.1 GHz Xeon vCPU for the load generator and server processes, no TLS, five runs at concurrency 1 and five at 32, with 10,000 measured requests after 3,000 warm-up requests.
Against a 0.054 millisecond passthrough control, identity-only verification added 0.201 milliseconds. Signed search with a receipt and log added 0.339 milliseconds. A delegated request added 0.494 milliseconds, and a paid request with voucher, receipt, and log added 0.650 milliseconds.
Throughput fell from about 15,892 requests per second for the control to 2,424 for signed search and 1,438 for the paid path. Those figures describe one shared core on loopback, not production Internet traffic. Native cryptography, worker threads, caching, and separate load generation could change them. The repository labels version 0.1 a prototype, with a local resolver, placeholder settlement vouchers, and an unanchored receipt log.
What a site can borrow now
The proposal is not an adopted web standard. Its grammar, delegation format, receipt format, and settlement integration remain open. The Web Bot Auth document is an Internet-Draft and may change before it becomes an RFC.
Publishers can put machine-use rules at the origin, authenticate operators instead of trusting user-agent strings, and sign request intent so proxies cannot rewrite it. Subscriber delegation should bind to one operator with narrow scope. Each delivery should record the terms that governed it. Contractual promises belong apart from controls the server actually enforces.
This also sharpens the policy problem discussed in Open weights do not erase the EU AI Act's training-summary duty. Machine-readable reservations matter only if operators can find them, interpret them consistently, and later show what they did.
An agent built through api.ish.chat can keep identity, user authority, declared purpose, and payment as separate credentials. A publisher using ish.chat cannot force every bot to behave. It can give signed clients access that anonymous scrapers do not receive, then keep evidence of the exchange.
