Cited AI crawler registry (training, search or user fetch) plus robots.txt checks. No fetching.
Maintain this server? Claim it
Call this server's tools from your browser. Nothing to install.
Requests go through mcptry because browsers can't call most MCP servers directly. Header values are sent to the server for that request only and are never stored or logged.
Tool list from our last check. Connect live to refresh it.
Match a robots.txt file the caller pastes against Botcard's AI crawler tokens, using RFC 9309. Each row is allow or disallow plus that token's vendor stance. This tool does not fetch any website. Do not send url, host, target, website, robots_url, or fetch.
The robots.txt file contents. Not a URL.
Card ids to check, such as gptbot. Omit to check every card.
Path to test, starting with /. Defaults to /.
Match a robots.txt file the caller pastes against Botcard's AI crawler tokens, using RFC 9309. Each row is allow or disallow plus that token's vendor stance. This tool does not fetch any website. Do not send url, host, target, website, robots_url, or fetch.
| Argument | Type | Description |
|---|---|---|
| ids | array | Card ids to check, such as gptbot. Omit to check every card. |
| path | string | Path to test, starting with /. Defaults to /. |
| robots_txt* | string | The robots.txt file contents. Not a URL. |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"robots_txt"
],
"properties": {
"ids": {
"type": "array",
"items": {
"type": "string"
},
"description": "Card ids to check, such as gptbot. Omit to check every card."
},
"path": {
"type": "string",
"description": "Path to test, starting with /. Defaults to /."
},
"robots_txt": {
"type": "string",
"description": "The robots.txt file contents. Not a URL."
}
},
"additionalProperties": false
}Return one card by id, including the vendor source URL and the date it was read. This tool does not fetch any website. Do not send url, host, target, website, robots_url, or fetch.
| Argument | Type | Description |
|---|---|---|
| id* | string | Card id, such as gptbot or google-extended. |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"id"
],
"properties": {
"id": {
"type": "string",
"description": "Card id, such as gptbot or google-extended."
}
},
"additionalProperties": false
}List the AI crawler robots.txt tokens in this dataset. Each row names the vendor page the summary was checked against. This tool does not fetch any website. Do not send url, host, target, website, robots_url, or fetch.
No arguments.
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {},
"additionalProperties": false
}Write a robots.txt from this dataset for the kinds you allow or block. Each group cites the vendor page. effect request_only means the vendor does not say the crawler obeys that line. control_token is not training. The file has no User-agent: * group. This tool does not fetch any website. Do not send url, host, target, website, robots_url, or fetch.
| Argument | Type | Description |
|---|---|---|
| path | string | Path the Allow or Disallow names. Defaults to /. |
| allow | array | Kinds to Allow, such as search. |
| block | array | Kinds to Disallow, such as training. |
| policy | string | Same choice in words, such as "allow search, block training". |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"path": {
"type": "string",
"description": "Path the Allow or Disallow names. Defaults to /."
},
"allow": {
"type": "array",
"items": {
"enum": [
"training",
"search",
"user_fetch",
"ads_check",
"control_token",
"owner_requested",
"open_crawl",
"safety",
"preview",
"other"
],
"type": "string"
},
"description": "Kinds to Allow, such as search."
},
"block": {
"type": "array",
"items": {
"enum": [
"training",
"search",
"user_fetch",
"ads_check",
"control_token",
"owner_requested",
"open_crawl",
"safety",
"preview",
"other"
],
"type": "string"
},
"description": "Kinds to Disallow, such as training."
},
"policy": {
"type": "string",
"description": "Same choice in words, such as \"allow search, block training\"."
}
},
"additionalProperties": false
}