Synthetic checks, nightly regression replay and model-drift alerts for AI agents
Maintain this server? Claim it
Call this server's tools from your browser. Nothing to install.
Bearer ak_live_… — create one at /dashboard/keys. Omit it to use the free read-only tools (agentcheck_get_status, agentcheck_get_pricing).
Requests go through mcptry because browsers can't call most MCP servers directly. Header values are sent to the server for that request only and are never stored or logged.
Tool list from our last check. Connect live to refresh it.
Add a check to one of your targets. A check runs an input (http path/prompt, mcp tool call, a2a message) on a schedule and judges the answer with a golden: exact, contains, regex and json_schema cost nothing; rubric and baseline use the LLM judge (baseline = same outcome as the last known-good answer). Returns the check id. Requires an API key.
what to run: http (GET path or POST prompt), mcp_tools_list, mcp_tool_call (tool + args), a2a_task (message)
short name shown on the status page
how the answer is judged. exact / contains / regex / json_schema are free and deterministic; rubric and baseline call the LLM judge, and baseline compares against the last known-good answer.
id from agentcheck_create_target or GET /api/v1/targets
http: { path?, method?, body?, prompt? } · mcp_tool_call: { tool, args } · a2a_task: { message }
seconds between runs; the tier's interval is the floor (Free hourly, Starter+ 5 min)
This tool isn't marked read-only, so it may change something on the server or in an account you connected.
Add a check to one of your targets. A check runs an input (http path/prompt, mcp tool call, a2a message) on a schedule and judges the answer with a golden: exact, contains, regex and json_schema cost nothing; rubric and baseline use the LLM judge (baseline = same outcome as the last known-good answer). Returns the check id. Requires an API key.
| Argument | Type | Description |
|---|---|---|
| kind* | string | what to run: http (GET path or POST prompt), mcp_tools_list, mcp_tool_call (tool + args), a2a_task (message) |
| name* | string | short name shown on the status page |
| input | object | http: { path?, method?, body?, prompt? } · mcp_tool_call: { tool, args } · a2a_task: { message } |
| golden* | object | how the answer is judged. exact / contains / regex / json_schema are free and deterministic; rubric and baseline call the LLM judge, and baseline compares against the last known-good answer. |
| targetId* | string | id from agentcheck_create_target or GET /api/v1/targets |
| intervalSec | integer | seconds between runs; the tier's interval is the floor (Free hourly, Starter+ 5 min) |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"name",
"kind",
"golden",
"targetId"
],
"properties": {
"kind": {
"enum": [
"http",
"mcp_tools_list",
"mcp_tool_call",
"a2a_task",
"custom"
],
"type": "string",
"description": "what to run: http (GET path or POST prompt), mcp_tools_list, mcp_tool_call (tool + args), a2a_task (message)"
},
"name": {
"type": "string",
"maxLength": 120,
"minLength": 1,
"description": "short name shown on the status page"
},
"input": {
"type": "object",
"default": {},
"description": "http: { path?, method?, body?, prompt? } · mcp_tool_call: { tool, args } · a2a_task: { message }",
"propertyNames": {
"type": "string"
},
"additionalProperties": {}
},
"golden": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"enum": [
"exact",
"contains",
"regex",
"json_schema",
"rubric",
"baseline"
],
"type": "string",
"description": "how the answer is judged; exact/contains/regex/json_schema cost nothing, rubric/baseline use the LLM judge"
},
"tools": {
"type": "array",
"items": {
"type": "string"
},
"description": "expected tool-call sequence (replay checks)"
},
"value": {
"type": "string",
"description": "exact / contains: the expected text"
},
"rubric": {
"type": "string",
"description": "rubric: what a passing answer must do, in plain language"
},
"schema": {
"type": "object",
"description": "json_schema: JSON Schema (draft 2020-12 subset)",
"propertyNames": {
"type": "string"
},
"additionalProperties": {}
},
"pattern": {
"type": "string",
"description": "regex: pattern source without slashes"
},
"expected": {
"type": "string",
"description": "rubric: a reference answer, when there is one"
},
"caseInsensitive": {
"type": "boolean"
}
},
"description": "how the answer is judged. exact / contains / regex / json_schema are free and deterministic; rubric and baseline call the LLM judge, and baseline compares against the last known-good answer."
},
"targetId": {
"type": "string",
"minLength": 1,
"description": "id from agentcheck_create_target or GET /api/v1/targets"
},
"intervalSec": {
"type": "integer",
"default": 3600,
"maximum": 604800,
"minimum": 60,
"description": "seconds between runs; the tier's interval is the floor (Free hourly, Starter+ 5 min)"
}
},
"additionalProperties": false
}When does the API version this code pins stop working, and what happens then? Pass provider and version pairs as your code pins them (shopify 2025-10, meta v21.0, google-ads v22, hubspot v1, twilio-flex 2.16.2, twilio-notify v1, keap xml-rpc, exchange-online ews, stripe 2024-06-20). For each, returns every dated end the provider's own versioning, changelog or retirement page lists: status, end date (and whether the provider gives only a month), what happens after it (error, silent fall-forward to another version, app delisting, or unsupported), whether the failure is silent, the replacement version, the source URL with the date it was read, and a page with the details. Where the provider's own pages give different dates for the same end, other_dates lists each one with its page and date_note says which the counts run to. A pair with no entry comes back with the reason: older than every listed version, no end date published yet, or a provider that publishes none (Stripe). Covers Shopify, HubSpot, Meta Graph and Marketing APIs, Google Ads and Content API for Shopping, Twilio Flex UI and Notify, Thryv's Keap XML-RPC, and Exchange Web Services (EWS) in Exchange Online. No API key needed.
| Argument | Type | Description |
|---|---|---|
| pins* | array | provider and version pairs, e.g. [{"provider": "shopify", "version": "2025-10"}] |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"pins"
],
"properties": {
"pins": {
"type": "array",
"items": {
"type": "object",
"required": [
"provider",
"version"
],
"properties": {
"api": {
"type": "string",
"maxLength": 60,
"minLength": 1,
"description": "optional, to narrow one provider to one API: admin or storefront (Shopify), graph-api or marketing-api (Meta), ads-api or content-api (Google), pipelines, rest or developer-platform (HubSpot), flex-ui or notify (Twilio), exchange-online (Microsoft); omit it to get every API of that provider with this version"
},
"version": {
"type": "string",
"maxLength": 60,
"minLength": 1,
"description": "the version as the code pins it: 2025-10, v21.0, v22, v1, 2025.2, 2.16.2, xml-rpc, ews, 2024-06-20"
},
"provider": {
"type": "string",
"maxLength": 60,
"minLength": 1,
"description": "the provider as you would name it: shopify, hubspot, meta (or facebook), google-ads, content-api, twilio-flex, twilio-notify, keap (or thryv, infusionsoft), ews (or exchange-online, microsoft), stripe"
}
},
"additionalProperties": false
},
"maxItems": 50,
"minItems": 1,
"description": "provider and version pairs, e.g. [{\"provider\": \"shopify\", \"version\": \"2025-10\"}]"
}
},
"additionalProperties": false
}Enroll something to monitor: an http endpoint (JSON or OpenAI-style chat), a remote MCP server (Streamable HTTP url) or an A2A agent (origin with /.well-known/agent-card.json). Pass `checks` to create checks in the same call (POST /api/v1/probe proposes three). Returns the target id, the public status page, the badge SVG URL and a README snippet. The first check runs on the next scheduled tick, within five minutes; call agentcheck_run_now to run immediately. Free tier: 1 target, hourly; Starter+: 5-minute checks. Requires an API key.
| Argument | Type | Description |
|---|---|---|
| url | string | endpoint URL (http/mcp) or the agent's origin (a2a) |
| auth | any | credential the prober sends to your endpoint: {type:'bearer',token} or {type:'header',name,value}. Stored encrypted and never returned or logged. |
| kind* | string | http (JSON or chat endpoint), mcp (Streamable HTTP url, or npx/uvx package spec), a2a (agent card) |
| name | string | display name; defaults to the host |
| slug | string | URL slug for /<owner>/<slug>; defaults to a slug of the name |
| alerts | object | where to send an incident when a check starts failing. Any combination; omit it and incidents are visible only on the status page. |
Machine-readable pricing for agentcheck: tiers with monthly USD price, target limits, check interval and features, plus per-run add-ons. Same data as /pricing.json. No API key needed.
No arguments.
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {},
"additionalProperties": false
}Current status of a public monitored target: overall state, uptime over 24h/7d/30d, last check time, last nightly scores, open incidents and the badge/status URLs. Use the owner (GitHub login) and target slug from the status page URL https://agentwares-agentcheck.vercel.app/<owner>/<slug>. No API key needed.
| Argument | Type | Description |
|---|---|---|
| slug* | string | target slug, e.g. demo |
| owner* | string | GitHub login of the target's owner, e.g. demo |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"owner",
"slug"
],
"properties": {
"slug": {
"type": "string",
"minLength": 1,
"description": "target slug, e.g. demo"
},
"owner": {
"type": "string",
"minLength": 1,
"description": "GitHub login of the target's owner, e.g. demo"
}
},
"additionalProperties": false
}Open and recent incidents across your targets (or one target): when they opened/closed, the failing check and the cause. Requires an API key.
| Argument | Type | Description |
|---|---|---|
| targetId | string | narrow to one target; omit for incidents across every target on the account |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"targetId": {
"type": "string",
"description": "narrow to one target; omit for incidents across every target on the account"
}
},
"additionalProperties": false
}Is this model retiring? Pass model IDs as your code or config names them (gpt-5-mini, claude-sonnet-4-5-20250929, gemini-2.5-flash, openai/o3-mini). For each, returns every announced shutdown on OpenAI's, Anthropic's and Google's own deprecation pages (OpenAI API, Claude API, Gemini API, Vertex AI): status, shutdown date, recommended replacement, the replacement's list-price ratio per token, the API changes the provider documents, the source URL with the date it was read, and a page with the details. retiring: false means no shutdown for that ID is on those pages. Language models only. No API key needed.
| Argument | Type | Description |
|---|---|---|
| models* | array | model IDs exactly as your code or config names them, e.g. ["gpt-5-mini", "claude-sonnet-4-5"]; router prefixes such as openai/ or models/ are ignored, and fine-tuned IDs (ft:…) match their base model |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"models"
],
"properties": {
"models": {
"type": "array",
"items": {
"type": "string",
"maxLength": 200,
"minLength": 1
},
"maxItems": 50,
"minItems": 1,
"description": "model IDs exactly as your code or config names them, e.g. [\"gpt-5-mini\", \"claude-sonnet-4-5\"]; router prefixes such as openai/ or models/ are ignored, and fine-tuned IDs (ft:…) match their base model"
}
},
"additionalProperties": false
}Turn an imported or recorded trace into a replayable check whose golden is the recorded outcome (tool sequence + final-answer rubric). The check runs daily and in the nightly replay (Pro). Returns the check. Requires an API key.
| Argument | Type | Description |
|---|---|---|
| traceId* | string | trace id from agentcheck_record or POST /api/v1/targets/{id}/traces |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"traceId"
],
"properties": {
"traceId": {
"type": "string",
"minLength": 1,
"description": "trace id from agentcheck_record or POST /api/v1/targets/{id}/traces"
}
},
"additionalProperties": false
}Record one production interaction with your agent (the prompt or messages, the final answer, the tools it called, optionally the model) as a trace on a target. Call it from your agent or from a proxy in front of it after each task; promote a good trace with agentcheck_promote_trace to replay it nightly and catch regressions. Requires an API key.
| Argument | Type | Description |
|---|---|---|
| name | string | short label, e.g. 'refund for order A-1029' |
| input* | any | the prompt, the messages array, or an object with prompt/messages |
| model | string | the model that produced this answer, so drift re-runs can compare across models |
| output* | string | the agent's final answer |
| targetId* | string | id from agentcheck_create_target or GET /api/v1/targets |
| toolCalls | array | tool calls in order |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"targetId",
"input",
"output"
],
"properties": {
"name": {
"type": "string",
"maxLength": 120,
"description": "short label, e.g. 'refund for order A-1029'"
},
"input": {
"anyOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "object",
"required": [
"role",
"content"
],
"properties": {
"role": {
"type": "string"
},
"content": {
"type": "string"
}
}
}
},
{
"type": "object",
"propertyNames": {
"type": "string"
},
"additionalProperties": {}
}
],
"description": "the prompt, the messages array, or an object with prompt/messages"
},
"model": {
"type": "string",
"description": "the model that produced this answer, so drift re-runs can compare across models"
},
"output": {
"type": "string",
"minLength": 1,
"description": "the agent's final answer"
},
"targetId": {
"type": "string",
"minLength": 1,
"description": "id from agentcheck_create_target or GET /api/v1/targets"
},
"toolCalls": {
"type": "array",
"items": {
"type": "object",
"required": [
"name"
],
"properties": {
"args": {},
"name": {
"type": "string"
}
}
},
"description": "tool calls in order"
}
},
"additionalProperties": false
}Run every check of one of your targets immediately (outside the schedule) and return pass/fail per check with latency, judge cost and any incident opened or closed. Use it right after enrolling, or to confirm a fix. Requires an API key.
| Argument | Type | Description |
|---|---|---|
| targetId* | string | target id |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"targetId"
],
"properties": {
"targetId": {
"type": "string",
"minLength": 1,
"description": "target id"
}
},
"additionalProperties": false
}Read a public Atlassian Statuspage page once: its name, overall status, components (id, name, status, group) and unresolved incidents, from the page's own /api/v2/summary.json. Pass any address on the page: status.example.com, https://status.example.com/history or the JSON URL. Use it before POST /api/v1/statuspage, which turns the components you give a URL into a live agentcheck status page checked by agentcheck itself, with no signup; the answer carries that request's body. Same read as /statuspage/preview: only public addresses, 5-second timeout, 512 KB cap. Private pages cannot be read. No API key needed.
| Argument | Type | Description |
|---|---|---|
| url* | string | the status page's address, e.g. https://status.example.com or its summary.json |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"url"
],
"properties": {
"url": {
"type": "string",
"maxLength": 2048,
"minLength": 1,
"description": "the status page's address, e.g. https://status.example.com or its summary.json"
}
},
"additionalProperties": false
}| checks |
| array |
| checks to create right away (the probe proposes three) |
| format | string | http: openai_chat (POST /chat/completions body), json (raw POST), get (plain fetch) |
| headers | object | extra request headers |
| isPublic | boolean | public status page + badge (default true) |
| modelVar | string | the model your agent runs on (e.g. claude-sonnet-5); enables model-drift re-runs |
| corpusUrl | string | RAG targets: public corpus URL for the nightly groundedness scorer (Pro) |
| modelHeader | string | request header your endpoint accepts to override the model (drift re-runs try the new model) |
| packageSpec | string | mcp only: `npx @org/server` / `uvx server` — runs on the mcpcheck runner, not the scheduled checks |
| agentCardUrl | string | a2a: explicit agent card URL when it is not at /.well-known/agent-card.json |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"kind"
],
"properties": {
"url": {
"type": "string",
"format": "uri",
"description": "endpoint URL (http/mcp) or the agent's origin (a2a)"
},
"auth": {
"anyOf": [
{
"type": "object",
"required": [
"type",
"token"
],
"properties": {
"type": {
"type": "string",
"const": "bearer"
},
"token": {
"type": "string"
}
}
},
{
"type": "object",
"required": [
"type",
"name",
"value"
],
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string",
"const": "header"
},
"value": {
"type": "string"
}
}
}
],
"description": "credential the prober sends to your endpoint: {type:'bearer',token} or {type:'header',name,value}. Stored encrypted and never returned or logged."
},
"kind": {
"enum": [
"http",
"mcp",
"a2a"
],
"type": "string",
"description": "http (JSON or chat endpoint), mcp (Streamable HTTP url, or npx/uvx package spec), a2a (agent card)"
},
"name": {
"type": "string",
"maxLength": 80,
"minLength": 1,
"description": "display name; defaults to the host"
},
"slug": {
"type": "string",
"pattern": "^[a-z0-9][a-z0-9-]{0,62}$",
"description": "URL slug for /<owner>/<slug>; defaults to a slug of the name"
},
"alerts": {
"type": "object",
"properties": {
"emails": {
"type": "array",
"items": {
"type": "string",
"format": "email",
"pattern": "^(?!\\.)(?!.*\\.\\.)([A-Za-z0-9_'+\\-\\.]*)[A-Za-z0-9_+-]@([A-Za-z0-9][A-Za-z0-9\\-]*\\.)+[A-Za-z]{2,}$"
}
},
"slackWebhookUrl": {
"type": "string",
"format": "uri"
},
"discordWebhookUrl": {
"type": "string",
"format": "uri"
}
},
"description": "where to send an incident when a check starts failing. Any combination; omit it and incidents are visible only on the status page."
},
"checks": {
"type": "array",
"items": {
"type": "object",
"required": [
"name",
"kind",
"golden"
],
"properties": {
"kind": {
"enum": [
"http",
"mcp_tools_list",
"mcp_tool_call",
"a2a_task",
"custom"
],
"type": "string",
"description": "what to run: http (GET path or POST prompt), mcp_tools_list, mcp_tool_call (tool + args), a2a_task (message)"
},
"name": {
"type": "string",
"maxLength": 120,
"minLength": 1,
"description": "short name shown on the status page"
},
"input": {
"type": "object",
"default": {},
"description": "http: { path?, method?, body?, prompt? } · mcp_tool_call: { tool, args } · a2a_task: { message }",
"propertyNames": {
"type": "string"
},
"additionalProperties": {}
},
"golden": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"enum": [
"exact",
"contains",
"regex",
"json_schema",
"rubric",
"baseline"
],
"type": "string",
"description": "how the answer is judged; exact/contains/regex/json_schema cost nothing, rubric/baseline use the LLM judge"
},
"tools": {
"type": "array",
"items": {
"type": "string"
},
"description": "expected tool-call sequence (replay checks)"
},
"value": {
"type": "string",
"description": "exact / contains: the expected text"
},
"rubric": {
"type": "string",
"description": "rubric: what a passing answer must do, in plain language"
},
"schema": {
"type": "object",
"description": "json_schema: JSON Schema (draft 2020-12 subset)",
"propertyNames": {
"type": "string"
},
"additionalProperties": {}
},
"pattern": {
"type": "string",
"description": "regex: pattern source without slashes"
},
"expected": {
"type": "string",
"description": "rubric: a reference answer, when there is one"
},
"caseInsensitive": {
"type": "boolean"
}
},
"description": "how the answer is judged. exact / contains / regex / json_schema are free and deterministic; rubric and baseline call the LLM judge, and baseline compares against the last known-good answer."
},
"intervalSec": {
"type": "integer",
"default": 3600,
"maximum": 604800,
"minimum": 60,
"description": "seconds between runs; the tier's interval is the floor (Free hourly, Starter+ 5 min)"
}
}
},
"maxItems": 20,
"description": "checks to create right away (the probe proposes three)"
},
"format": {
"enum": [
"openai_chat",
"json",
"get"
],
"type": "string",
"description": "http: openai_chat (POST /chat/completions body), json (raw POST), get (plain fetch)"
},
"headers": {
"type": "object",
"description": "extra request headers",
"propertyNames": {
"type": "string"
},
"additionalProperties": {
"type": "string"
}
},
"isPublic": {
"type": "boolean",
"default": true,
"description": "public status page + badge (default true)"
},
"modelVar": {
"type": "string",
"description": "the model your agent runs on (e.g. claude-sonnet-5); enables model-drift re-runs"
},
"corpusUrl": {
"type": "string",
"format": "uri",
"description": "RAG targets: public corpus URL for the nightly groundedness scorer (Pro)"
},
"modelHeader": {
"type": "string",
"description": "request header your endpoint accepts to override the model (drift re-runs try the new model)"
},
"packageSpec": {
"type": "string",
"description": "mcp only: `npx @org/server` / `uvx server` — runs on the mcpcheck runner, not the scheduled checks"
},
"agentCardUrl": {
"type": "string",
"format": "uri",
"description": "a2a: explicit agent card URL when it is not at /.well-known/agent-card.json"
}
},
"additionalProperties": false
}