RobotsGate

Working · 1.2 s

dev.workers.mike-tusa.robotsgate/robotsgatev0.3.1 Domain verified

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

Developer Tools Marketing & SEO

Website Registry entry

Maintain this server? Claim it

Try it

Call this server's tools from your browser. Nothing to install.

https://robotsgate.mike-tusa.workers.dev/mcp

Authorization (optional)

Optional. 'Bearer <RobotsGate Pro license key>' for higher limits. Works without a key.

Requests go through mcptry because browsers can't call most MCP servers directly. Header values are sent to the server for that request only and are never stored or logged.

Tool list from our last check. Connect live to refresh it.

check_site Read-only

Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check.

urlstring, required

Site URL or bare hostname, e.g. example.com. Only the scheme and host are used.

pathstring

URL path to test, starting with "/" (default "/").

Tools (4)

check_site

read-only

Argument	Type	Description
url*	string	Site URL or bare hostname, e.g. example.com. Only the scheme and host are used.
path	string	URL path to test, starting with "/" (default "/").

Input schema (JSON)

{
  "type": "object",
  "required": [
    "url"
  ],
  "properties": {
    "url": {
      "type": "string",
      "maxLength": 2048,
      "minLength": 1,
      "description": "Site URL or bare hostname, e.g. example.com. Only the scheme and host are used."
    },
    "path": {
      "type": "string",
      "maxLength": 2048,
      "description": "URL path to test, starting with \"/\" (default \"/\")."
    }
  },
  "additionalProperties": false
}

generate_robots

Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.

read-only

Argument	Type	Description
date	string	Date for the header comment (YYYY-MM-DD). Default: today (UTC).
agents	object	Per-token overrides, e.g. {"GPTBot": "block"}. Tokens: letters, digits, "_", "." and "-", up to 64 characters (not "*"). At most 100.
default	string	Policy for "User-agent: *" (default allow).
sitemap	string	A single sitemap URL (alternative to sitemaps).
sitemaps	array	Absolute http(s) sitemap URLs. At most 20.
categories	object	Action per crawler category. Omitted categories get no group of their own.
allow_paths	array	Paths to allow explicitly. Each path starts with "/" or "*" and has no whitespace or "#". At most 100.
custom_rules	string	Extra robots.txt lines appended after validation (max 20000 bytes).
disallow_paths	array	Paths to disallow for "" and for allowed AI groups. Each path starts with "/" or "" and has no whitespace or "#". At most 100.

Input schema (JSON)

{
  "type": "object",
  "properties": {
    "date": {
      "type": "string",
      "pattern": "^\\d{4}-\\d{2}-\\d{2}$",
      "maxLength": 10,
      "description": "Date for the header comment (YYYY-MM-DD). Default: today (UTC)."
    },
    "agents": {
      "type": "object",
      "description": "Per-token overrides, e.g. {\"GPTBot\": \"block\"}. Tokens: letters, digits, \"_\", \".\" and \"-\", up to 64 characters (not \"*\"). At most 100.",
      "maxProperties": 100,
      "propertyNames": {
        "pattern": "^[A-Za-z0-9_.-]{1,64}$"
      },
      "additionalProperties": {
        "enum": [
          "allow",
          "block",
          "omit"
        ],
        "type": "string"
      }
    },
    "default": {
      "enum": [
        "allow",
        "block"
      ],
      "type": "string",
      "description": "Policy for \"User-agent: *\" (default allow)."
    },
    "sitemap": {
      "type": "string",
      "maxLength": 2048,
      "description": "A single sitemap URL (alternative to sitemaps)."
    },
    "sitemaps": {
      "type": "array",
      "items": {
        "type": "string",
        "maxLength": 2048
      },
      "maxItems": 20,
      "description": "Absolute http(s) sitemap URLs. At most 20."
    },
    "categories": {
      "type": "object",
      "properties": {
        "other": {
          "enum": [
            "allow",
            "block",
            "omit"
          ],
          "type": "string"
        },
        "search": {
          "enum": [
            "allow",
            "block",
            "omit"
          ],
          "type": "string"
        },
        "dataset": {
          "enum": [
            "allow",
            "block",
            "omit"
          ],
          "type": "string"
        },
        "training": {
          "enum": [
            "allow",
            "block",
            "omit"
          ],
          "type": "string"
        },
        "user_fetch": {
          "enum": [
            "allow",
            "block",
            "omit"
          ],
          "type": "string"
        }
      },
      "description": "Action per crawler category. Omitted categories get no group of their own.",
      "additionalProperties": false
    },
    "allow_paths": {
      "type": "array",
      "items": {
        "type": "string",
        "maxLength": 512,
        "minLength": 1
      },
      "maxItems": 100,
      "description": "Paths to allow explicitly. Each path starts with \"/\" or \"*\" and has no whitespace or \"#\". At most 100."
    },
    "custom_rules": {
      "type": "string",
      "maxLength": 20000,
      "description": "Extra robots.txt lines appended after validation (max 20000 bytes)."
    },
    "disallow_paths": {
      "type": "array",
      "items": {
        "type": "string",
        "maxLength": 512,
        "minLength": 1
      },
      "maxItems": 100,
      "description": "Paths to disallow for \"*\" and for allowed AI groups. Each path starts with \"/\" or \"*\" and has no whitespace or \"#\". At most 100."
    }
  },
  "additionalProperties": false
}

list_crawlers

Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers.

read-only

Argument	Type	Description
category	string	Optional: only crawlers in this category.

Input schema (JSON)

{
  "type": "object",
  "properties": {
    "category": {
      "enum": [
        "training",
        "dataset",
        "search",
        "user_fetch",
        "other"
      ],
      "type": "string",
      "description": "Optional: only crawlers in this category."
    }
  },
  "additionalProperties": false
}

validate_robots

Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site.

read-only

Argument	Type	Description
path	string	URL path to test, starting with "/" (default "/").
robots_txt*	string	The robots.txt file content.

Input schema (JSON)

{
  "type": "object",
  "required": [
    "robots_txt"
  ],
  "properties": {
    "path": {
      "type": "string",
      "maxLength": 2048,
      "description": "URL path to test, starting with \"/\" (default \"/\")."
    },
    "robots_txt": {
      "type": "string",
      "maxLength": 50000,
      "description": "The robots.txt file content."
    }
  },
  "additionalProperties": false
}