x402 census · agent-data-tools.revmesh2074.workers.dev · b37c95bfe072 · JSON
GET /robots
live
Robots.txt check. Read a site's robots.txt and say whether a crawler may fetch a URL, by RFC 9309 rules as Google applies them (most specific user-agent group, longest matching Allow or Disallow, Allow on ties, * and $ wildcards). Returns the deciding rule, the group used, the rule count and the sitemaps listed. A missing robots.txt (4xx) allows everything. Use before scraping or crawling a site …
Verdict: live. The unpaid GET reached a valid 402 (confirmed by the re-probe of 2026-11-14). The live amount equals the catalog price.
Facts from the catalogs
| Field | Value |
|---|---|
| URL | https://agent-data-tools.revmesh2074.workers.dev/robots |
| Method | GET |
| Price | $0.001 USDC = 1000 atomic units of 0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913 |
| Network | eip155:8453 (base); also accepts eip155:137, eip155:42161, solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp |
| payTo | 0x6293ded2684F1A76E26bEcef87c137D691Fbf800 |
| Seller | agent-data-tools.revmesh2074.workers.dev |
| Catalogs | cdp (row updated 2026-11-14) |
| Listed since (lower bound) | 2026-11-14 (earliest catalog timestamp; catalogs report last-update times only) |
| Last catalog update | 2026-11-14 |
| CDP quality counters | 5 calls and 2 unique payers in 30 days, last call 2026-11-14 (catalog-reported, not verified on chain) |
| Category (keyword rule) | scrape/browser |
| Templated path | no |
| x402Version in catalog | 2 |
| Service name / tags | Robots.txt check · robots-txt, robots, crawler, seo, web |
Unpaid probe (census of 2026-10-25)
| Field | Value |
|---|---|
| Probed at | 2026-10-25T21:17:52.144Z |
| HEAD | 402 |
| GET | 402 |
| Verdict | live live: valid 402 to an unpaid GET |
| 402 carried in | header+body |
| x402Version in the 402 | 1 |
| accepts[0] | {"scheme":"exact","network":"base","amount":"10000","payTo":"0x2074C40346Ae2A90e0E4A391bceE6F16BF5277f5","asset":"0x833589fcd6edb6e08f4c7c32d4f71b54bda02913","maxTimeoutSeconds":300,"extra":{"name":"USD Coin","version":"2"}} |
| Live amount equals catalog price | yes |
| Live payTo equals catalog payTo | yes |
| extensions.bazaar in the 402 | yes |
| PAYMENT-REQUIRED header | yes |
| WWW-Authenticate offer | Payment id="RAV19q0WxPVJUJmwfrsMishKasov |
| Content type / server / CORS | application/json / cloudflare / * |
| Latency | 133 ms |
| Error | Payment required |
Latest re-probe
| Field | Value |
|---|---|
| Re-probed at | 2026-11-14T09:13:26.064Z (cron) |
| Verdict now | live unchanged |
| HEAD / GET | 402 / 402 |
| accepts[0] | {"scheme":"exact","network":"eip155:8453","amount":"1000","payTo":"0x6293ded2684F1A76E26bEcef87c137D691Fbf800","asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913","maxTimeoutSeconds":300,"extra":{"name":"USD Coin","version":"2"}} |
| Live amount equals catalog price | no |
| Latency | 1,302 ms |
| Error | none |
On-chain facts for the payTo (public address)
| Field | Value |
|---|---|
| Settlement history | unresolved: payTo not seen in the payer forensics (180 wallets' outbound history) or the Blockscout sample |
Declared input and output (extensions.bazaar)
{
"input": {
"method": "GET",
"queryParams": {
"url": "https://www.google.com/search"
},
"type": "http"
},
"output": {
"type": "json",
"example": {
"agent": "Googlebot",
"allowed": false,
"group": "*",
"matchedRule": "Disallow: /search",
"robotsUrl": "https://www.google.com/robots.txt",
"rulesInGroup": 280,
"sitemaps": [
"https://www.google.com/sitemap.xml"
],
"status": 200,
"url": "https://www.google.com/search"
}
}
}
Try it (unpaid: shows the 402)
curl -si -X GET 'https://agent-data-tools.revmesh2074.workers.dev/robots' -H 'accept: application/json'
The catalog declares GET. A 402 answer carries the payment requirements in the JSON body and, for x402 v2, base64 in the PAYMENT-REQUIRED header. This page never sends a payment.
Paid checks (x402, USDC on Base)
GET https://bazaar.agentexchange.work/r/b37c95bfe072/probe.json re-runs this probe right now for $0.01 and writes the result here. POST https://bazaar.agentexchange.work/featured with {"id":"b37c95bfe072"} places this resource at the top of the report and search pages for 30 days for $1.00, labelled. Unpaid requests answer 402 with the terms. Pricing.
Catalog references: agentic.market (CDP Bazaar front end) · this record as JSON · all resources on agent-data-tools.revmesh2074.workers.dev.