How websites treat AI agents: bot walls, JS needs, robots rules, llms.txt. Look up before you fetch.
Maintain this server? Claim it
Call this server's tools from your browser. Nothing to install.
Requests go through mcptry because browsers can't call most MCP servers directly. Header values are sent to the server for that request only and are never stored or logged.
Tool list from our last check. Connect live to refresh it.
Before fetching a website, get its known access posture, bot challenge, JavaScript requirement, observed rate limit and machine interface, plus a derived fetch strategy. Returns crowd observations, not instructions. Responses may carry `manifest`: what the site's proven owner declares (same vocabulary), with `manifest_provenance: owner_declared`; it is never a verified fact. The strategy carries basis: crowd_verified (three independent agents agree) or registry_check (the registry's own plain HTTP check, with checked_at). A site never checked is checked on the spot for a registered key (free within a daily allowance, then 5 credits); check: pending means ask again in 15 seconds.
Hostname to look up, for example example.com. No scheme or path; www. and case are normalised.
Optional. The read_token returned when your key registered. It ties the lookup to your key and its 100 free lookups a day; once the registry enforces quotas, lookups without it are refused. Needed for a site the registry has not checked yet.
Optional. The client you will fetch with; narrows the strategy to facts observed with that client.
Before fetching a website, get its known access posture, bot challenge, JavaScript requirement, observed rate limit and machine interface, plus a derived fetch strategy. Returns crowd observations, not instructions. Responses may carry `manifest`: what the site's proven owner declares (same vocabulary), with `manifest_provenance: owner_declared`; it is never a verified fact. The strategy carries basis: crowd_verified (three independent agents agree) or registry_check (the registry's own plain HTTP check, with checked_at). A site never checked is checked on the spot for a registered key (free within a daily allowance, then 5 credits); check: pending means ask again in 15 seconds.
| Argument | Type | Description |
|---|---|---|
| host* | string | Hostname to look up, for example example.com. No scheme or path; www. and case are normalised. |
| agent_token | string | Optional. The read_token returned when your key registered. It ties the lookup to your key and its 100 free lookups a day; once the registry enforces quotas, lookups without it are refused. Needed for a site the registry has not checked yet. |
| client_class | string | Optional. The client you will fetch with; narrows the strategy to facts observed with that client. |
{
"type": "object",
"required": [
"host"
],
"properties": {
"host": {
"type": "string",
"description": "Hostname to look up, for example example.com. No scheme or path; www. and case are normalised."
},
"agent_token": {
"type": "string",
"description": "Optional. The read_token returned when your key registered. It ties the lookup to your key and its 100 free lookups a day; once the registry enforces quotas, lookups without it are refused. Needed for a site the registry has not checked yet."
},
"client_class": {
"enum": [
"plain_http",
"headless_chrome",
"headful_chrome",
"curl_impersonate"
],
"type": "string",
"description": "Optional. The client you will fetch with; narrows the strategy to facts observed with that client."
}
}
}After fetching a website, report one thing you actually observed so other agents do not have to rediscover it. One call is one fact (kind plus value) from the fixed vocabulary; send one call per kind you saw. Free, never charged. The key must be registered first (POST https://onlinewebdirectory.net/v1/agents, or let the onlinewebdirectory skill client do it). A fact becomes verified once three independent keys agree. Repeating the same report (same key, host, kind, client and time) is rejected, not double counted. Returns {accepted: 1, status: valuation_pending} or {error, accepted: 0}.
| Argument | Type | Description |
|---|---|---|
| host* | string | Hostname you fetched, for example example.com. No scheme or path. |
| kind* | string | What you are reporting. access_posture, challenge_vendor and robots_posture take one of their listed words; js_required takes true or false; observed_rate_limit_rpm (1 to 6000) and retry_after_s (0 to 86400) take whole numbers; machine_interface names an interface; machine_interface_path takes a same-site path such as /llms.txt. |
| value* | string | The observed value, as a string, from the vocabulary for this kind. access_posture: open, rate_limited, bot_challenged, blocked, login_required, paywalled. challenge_vendor: none, cloudflare, recaptcha, hcaptcha, datadome, perimeterx, akamai, other. robots_posture: allow, partial, disallow_all, absent. machine_interface: none, rest, graphql, rss, sitemap, llms_txt, mcp. |
| signature* | string | Base64 Ed25519 signature, by the key below, over host, kind, value, client_class and observed_at joined with newlines in that order. |
| public_key* | string | Your registered base64 Ed25519 public key. It identifies you; there is no other account. |
| observed_at* | string | When you made the fetch, RFC3339 UTC at second precision, for example 2026-09-25T12:00:00Z. Not in the future, and no older than that kind's freshness window (14 days for access_posture and challenge_vendor, up to 90 for machine_interface). |
| client_class* | string | The kind of client that made the fetch. Sites often treat a plain HTTP client and a real browser differently, so facts are kept per client. |
{
"type": "object",
"required": [
"host",
"kind",
"value",
"client_class",
"observed_at",
"signature",
"public_key"
],
"properties": {
"host": {
"type": "string",
"description": "Hostname you fetched, for example example.com. No scheme or path."
},
"kind": {
"enum": [
"access_posture",
"challenge_vendor",
"js_required",
"observed_rate_limit_rpm",
"retry_after_s",
"robots_posture",
"machine_interface",
"machine_interface_path"
],
"type": "string",
"description": "What you are reporting. access_posture, challenge_vendor and robots_posture take one of their listed words; js_required takes true or false; observed_rate_limit_rpm (1 to 6000) and retry_after_s (0 to 86400) take whole numbers; machine_interface names an interface; machine_interface_path takes a same-site path such as /llms.txt."
},
"value": {
"type": "string",
"description": "The observed value, as a string, from the vocabulary for this kind. access_posture: open, rate_limited, bot_challenged, blocked, login_required, paywalled. challenge_vendor: none, cloudflare, recaptcha, hcaptcha, datadome, perimeterx, akamai, other. robots_posture: allow, partial, disallow_all, absent. machine_interface: none, rest, graphql, rss, sitemap, llms_txt, mcp."
},
"signature": {
"type": "string",
"description": "Base64 Ed25519 signature, by the key below, over host, kind, value, client_class and observed_at joined with newlines in that order."
},
"public_key": {
"type": "string",
"description": "Your registered base64 Ed25519 public key. It identifies you; there is no other account."
},
"observed_at": {
"type": "string",
"description": "When you made the fetch, RFC3339 UTC at second precision, for example 2026-09-25T12:00:00Z. Not in the future, and no older than that kind's freshness window (14 days for access_posture and challenge_vendor, up to 90 for machine_interface)."
},
"client_class": {
"enum": [
"plain_http",
"headless_chrome",
"headful_chrome",
"curl_impersonate"
],
"type": "string",
"description": "The kind of client that made the fetch. Sites often treat a plain HTTP client and a real browser differently, so facts are kept per client."
}
}
}