Robotbase Dev

Developer ecosystem trends: GitHub repos, Hugging Face models, npm packages, PyPI stats and Hacker News. When to use: choosing a library/model, seeing what developers are adopting, checking package health. Do not use: for general web search (web_search) or social hot lists (social_hot). Returns: {ecosystem, items:[{name, metric, value, url, description, source}], errors}.

Drive a real Chromium browser on one public page: click, type, press keys, scroll, wait for selectors, then return the resulting text (and optionally a screenshot and JSON XHR responses). When to use: the page needs interaction before its content appears (cookie banners, tabs, 'load more', form submit) and web_fetch/web_crawl cannot see it. Do not use: for plain reading (web_fetch) or bulk crawling (web_crawl); no login/paywall bypass. Returns: {ok, url, final_url, title, markdown, chars, elapsed_ms, screenshot?, xhr?, took_ms, _metering}. Max 10 actions; screenshot returns MCP image content. Metered: shares the web free bucket (100 calls/day per IP shared by all web tools), then 3 credits per call over x402.

Crawl one site's same-host pages starting from a URL (depth 1-2, at most 20 pages) and return each page's title, URL and readable text. When to use: you need several pages from one site (docs section, news index, product list) and web_map alone is not enough. Do not use: to mirror a whole site (hard cap 20 pages, same host), or for one page (web_fetch). Returns: {ok, url, final_url, count, pages:[{url, title, status, chars, content}], cached, took_ms, _metering}. Respects robots.txt and SSRF guards; partial page failures are skipped. Metered: shares the web free bucket (100 calls/day per IP shared by all web tools), then 3 credits per call over x402 - no API key.

Submit a same-host crawl as an asynchronous job and return a job_id immediately. When to use: you need more than a couple of pages and cannot hold a long HTTP request; submit here, then poll web_crawl_job until status is done, failed or cancelled. Do not use: for one page (web_fetch) or when you need the result in this call (web_crawl). Returns: {ok, job_id, status, url, depth, max_pages, max_chars, idempotent, poll_with}. Pass idempotency_key to retry safely (24h window returns the same job). Metered: shares the web free bucket (100 calls/day per IP), then 3 credits per submit over x402.

Check or cancel an asynchronous crawl job created by web_crawl_async. When to use: after web_crawl_async, poll with action=status until status is done/failed/cancelled; use action=cancel to stop a running job. Do not use: to start a crawl (web_crawl_async) or for a quick one-page read (web_fetch). Returns status: {ok, job_id, status: queued|running|done|failed|cancelled, progress:{pages_done,max_pages}, created_at, updated_at, result?, error?}. Free (not metered); rate-limited. Done result uses the same shape as web_crawl.

Fetch a page now and compare it with the previous snapshot of the same URL, returning whether it changed and a unified diff (no LLM). When to use: monitoring a page for updates - prices, docs, policy pages, SEO titles/meta; first call stores the baseline, later calls return added/removed lines. Do not use: for one-off reading (web_fetch) or structured fields (web_extract_schema). Returns: {ok, url, baseline, changed, diff, added_lines, removed_lines, previous_ts, current_ts, title, hash, chars, fetched_cached, took_ms, _metering}. fresh=true (default) adds a cache-busting query to bypass the 6h page cache; pass fresh=false to compare cached content. Snapshots are kept per URL in SQLite (latest only). Metered: shares the web free bucket (100 calls/day per IP), then 1 credit per call over x402.

Extract structured data from one public page: tables (headers + rows), JSON-LD blocks, and Open Graph/Twitter/canonical/language metadata. When to use: the page has tables or schema.org JSON-LD you need as data, without an LLM. Do not use: when you only need the page text (web_fetch) or a screenshot (web_screenshot). Returns: {ok, url, final_url, title, meta:{description,image,site_name,type,canonical,lang}, jsonld[], tables:[{caption,headers,rows}], links? , cached, took_ms, _metering}. Metered: shares the web free bucket (100 calls/day per IP shared by all web tools), then 1 credit per call over x402.

Extract a page's structured data into a named schema's common fields (no LLM): article, product, event, faq, recipe, course, organization, local_business, breadcrumb, dataset or job_posting. When to use: you need fields (title/author/date, price/currency/availability, event dates, FAQ Q&A, recipe ingredients, org address) rather than raw markdown or tables. Do not use: to get every JSON-LD block verbatim (web_metadata/web_extract) or to read text (web_fetch). Returns: {ok, url, final_url, schema, requested_schema, detected_types, matched, fields:{...}, sources:[jsonld|meta], jsonld_count, tables_count, took_ms, cached, _metering}. If no matching JSON-LD exists it falls back to Open Graph/meta fields and matched=false. Metered: shares the web free bucket (100 calls/day per IP), then 1 credit per call over x402.

Read an RSS or Atom feed and return its latest items. When to use: the user wants the latest posts/news from a site that publishes a feed, or you found a feed URL and want headlines plus links. Do not use: to read article text (use web_fetch) or to discover a site's URLs (use web_map). Returns: {ok, url, final_url, title, count, items:[{title, link, published, summary}], cached, took_ms, _metering}. Summary is HTML-stripped and capped at 500 characters. Metered: shares the web free bucket (100 calls/day per IP shared by all web tools), then 1 credit per call over x402 - there is no API key.

Fetch one public web page and return its readable text as clean markdown or plain text, with navigation, scripts, headers and footers stripped. When to use: you have a specific URL (from web_search or the user) and need what the page actually says - 'read this article', 'summarise this docs page', 'turn this URL into markdown'. Do not use: to crawl or mirror a site, to fetch a page behind a login or paywall, to read API/RPC endpoints, or for book text inside CyberLibrary (use cyberlibrary_get_chapter). Returns: {url, final_url, status, title, content, chars, truncated, fetched_at, source (direct|provider), cached, took_ms, _metering}. The response is capped at 12000 characters per call; raise max_chars (up to 20000) when you need more. Reads one public URL only, respects robots.txt, and does not bypass login or paywall. Direct fetch is tried first; pages blocked from this node are fetched through a provider and marked source=provider. Metered: all web tools share 100 free calls/day per IP (separate from the risk/library allowance), then 1 credit per call over x402 - there is no API key; the 402 answer carries accepts[].

Fetch up to 5 public URLs in one call and return each page's title, engine and a short snippet (first ~1200 characters), so an agent can triage several links quickly. When to use: search returned several candidate pages and you want a first pass over all of them; then call web_fetch on the ones worth reading in full. Do not use: when you need the complete text of a page (use web_fetch), or to crawl a site (use web_map). Returns: {ok, count, ok_count, results:[{url, ok, final_url, status, title, engine, chars, snippet, error?}], took_ms, _metering}. Partial failures do not fail the whole call. Metered: counts one call per URL against the shared web free bucket (50 URLs/day per IP together with web_fetch), then 1 credit per URL over x402 - max 5 URLs per call, no API key.

Discover the URLs of one site - sitemap first, then same-host links from the homepage. When to use: you need the list of pages a site exposes before fetching a few of them. Do not use: as a crawler (max 50 URLs, no page content) or to bypass robots.txt. Returns: {url, final_url, count, urls:[{url,source}], cached, took_ms, _metering}. Metered: shares the web free bucket (100 calls/day per IP shared by all web tools), then 1 credit per call over x402 - there is no API key; the 402 answer carries accepts[].

Extract a page's structured metadata: Open Graph, Twitter cards, canonical URL, language and JSON-LD. When to use: you need the title/description/author/image of a page, or JSON-LD structured data for a product/article, without reading the whole page. Do not use: when you need the page text (use web_fetch) or a screenshot (use web_screenshot). Returns: {ok, url, final_url, title, description, image, site_name, type, canonical, lang, jsonld[], cached, took_ms, _metering}. Metered: shares the web free bucket (100 calls/day per IP shared by all web tools), then 1 credit per call over x402 - there is no API key.

Delete one scheduled change monitor owned by the calling IP (its event history is removed too). When to use: the user no longer needs to watch a page, or the per-IP monitor limit was reached. Do not use: to pause a monitor temporarily - there is no pause; delete and re-create. Returns: {ok, deleted, monitor_id, note}; unknown or foreign ids return {ok:false,error:'monitor_not_found'} and delete nothing. Free (not metered); rate-limited to 20 calls/min per IP.

List the scheduled change monitors owned by the calling IP, with latest/next check time, last title or error, and the 5 most recent change events. When to use: after web_monitor_set, poll this instead of re-fetching the page; or to show the user what is currently being watched. Do not use: to create a monitor (web_monitor_set) or to compare one page now (web_diff). Returns: {ok, count, monitors:[{monitor_id,url,interval_minutes,active,created_ts,last_check_ts,next_check_ts,last_title,last_error,webhook_enabled,events_total,recent_events}], note}. Pass monitor_id to get that one monitor plus last_event.diff (the most recent detected change, up to 4000 chars) so you can read what changed without a webhook. Free (not metered); rate-limited to 20 calls/min per IP.

Create a scheduled change monitor for one public URL: the server re-fetches the page on an interval, stores a new snapshot and records a diff event whenever the readable text changes. When to use: the user wants to be told when a page changes (price, docs, policy page, job post, release notes) and cannot poll it themselves; also for handing off 'watch this for me'. Do not use: for a single comparison right now (web_diff) or when you only need the page today (web_fetch). Returns: {ok, monitor_id, url, status:active, interval_minutes, next_check_ts, webhook_enabled, webhook_secret?, webhook_signature?, hint}. The first check runs within ~60s to store the baseline; later changes show up in web_monitor_list.recent_events and, if requested, are POSTed to webhook_url. Webhook: https only, no redirects, 8s timeout, up to 2 retries (30s/120s); body JSON {monitor_id,url,ts,changed,added_lines,removed_lines,title,diff,event_id}; verify header 'X-RobotBase-Signature: sha256=<hex HMAC-SHA256 of the raw body using webhook_secret>'. Limits: 15-1440 minutes; max 5 monitors per IP; monitors expire after 7 days. Metered: 1 credit (shares the web free bucket of 100 calls/day per IP).

Capture a screenshot of one public web page as an image (JPEG) plus metadata. When to use: visual layout matters - a dashboard, chart, UI state, or 'what does this page look like'. Do not use: to read text (use web_fetch), for pages behind a login or paywall, or for bulk capture. Returns: image content (image/jpeg) plus {url, final_url, title, width, height, bytes, engine, elapsed_ms, _metering}. Rendered by a real Chromium browser, robots.txt-respecting, SSRF-guarded, max 3MB per image. Metered: shares the web free bucket (100 calls/day per IP shared by all twelve web tools), then 2 credits per call over x402 - there is no API key; the 402 answer carries accepts[].

Search the live web and get ranked results with title, URL, snippet and published date. When to use: an agent needs current facts, news or links that are not on chain - 'what is the BTC fee market saying today', 'recent Zcash governance proposals', 'find the docs page for X'. Use it before web_fetch, then fetch the one URL that looks authoritative. Do not use: for on-chain facts (use the BTC/ETH/XMR/ZEC/TRON tools) or for content inside a CyberLibrary book (use cyberlibrary_fulltext_search). Provider: self-hosted SearXNG (Bing/Baidu/360/Sogou/Yandex/Marginalia) first, then Serper, Tavily and cn.bing RSS as automatic fallbacks. Returns: {query, count, results:[{title,url,snippet,published}], provider (serper|tavily|bing_rss), cached, took_ms, _metering}. Results may be cached for up to 10 minutes; a cached result still counts as one call against the free allowance. Metered: all web tools share 100 free calls/day per IP (separate from the risk/library allowance), then 1 credit per call over x402 - there is no API key; the 402 answer carries accepts[]. max count is 10.