robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
Maintain this server? Claim it
Call this server's tools from your browser. Nothing to install.
Requests go through mcptry because browsers can't call most MCP servers directly. Header values are sent to the server for that request only and are never stored or logged.
Tool list from our last check. Connect live to refresh it.
AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.
Site or page URL
AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.
| Argument | Type | Description |
|---|---|---|
| url* | string | Site or page URL |
{
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string",
"examples": [
"https://www.theguardian.com/"
],
"description": "Site or page URL"
}
}
}Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.
| Argument | Type | Description |
|---|---|---|
| url* | string | Page URL to test |
| user_agent | string | Crawler product token, e.g. GPTBot or MyAgent. Default * |
{
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string",
"examples": [
"https://www.nytimes.com/section/technology"
],
"description": "Page URL to test"
},
"user_agent": {
"type": "string",
"examples": [
"GPTBot"
],
"description": "Crawler product token, e.g. GPTBot or MyAgent. Default *"
}
}
}Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.
| Argument | Type | Description |
|---|---|---|
| url* | string | Site URL (discovers the sitemap) or a sitemap URL |
{
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string",
"examples": [
"https://www.python.org/"
],
"description": "Site URL (discovers the sitemap) or a sitemap URL"
}
}
}