Real cost per AI result, LLM prices, switch costs, shutdown dates, A/B test sizes and judge checks.
Maintain this server? Claim it
Call this server's tools from your browser. Nothing to install.
Requests go through mcptry because browsers can't call most MCP servers directly. Header values are sent to the server for that request only and are never stored or logged.
Tool list from our last check. Connect live to refresh it.
Returns how many results each version of a prompt or model needs before a difference in how often results work first time is statistically real (two-proportion z-test, two-sided), the total across both versions, and how many days that takes at a given daily volume.
Share of version B's results that work first time, percent (above 0, below 100)
Share of version A's (the current prompt or model) results that work first time, percent (above 0, below 100)
Chance of detecting a real difference of this size, percent
Confidence level, percent. Two-sided.
Results per day across both versions, split evenly. Gives the number of days.
Pick your client. Values in capitals, like YOUR_TOKEN, are yours to fill in.
Run in a terminal
claude mcp add --transport http tools https://aiappsthatwork.com/mcp
Returns how many results each version of a prompt or model needs before a difference in how often results work first time is statistically real (two-proportion z-test, two-sided), the total across both versions, and how many days that takes at a given daily volume.
| Argument | Type | Description |
|---|---|---|
| powerPct | number | Chance of detecting a real difference of this size, percent |
| targetPct* | number | Share of version B's results that work first time, percent (above 0, below 100) |
| baselinePct* | number | Share of version A's (the current prompt or model) results that work first time, percent (above 0, below 100) |
| confidencePct | number | Confidence level, percent. Two-sided. |
| resultsPerDay | number | Results per day across both versions, split evenly. Gives the number of days. |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"baselinePct",
"targetPct"
],
"properties": {
"powerPct": {
"enum": [
80,
90
],
"type": "number",
"default": 80,
"description": "Chance of detecting a real difference of this size, percent"
},
"targetPct": {
"type": "number",
"description": "Share of version B's results that work first time, percent (above 0, below 100)",
"exclusiveMaximum": 100,
"exclusiveMinimum": 0
},
"baselinePct": {
"type": "number",
"description": "Share of version A's (the current prompt or model) results that work first time, percent (above 0, below 100)",
"exclusiveMaximum": 100,
"exclusiveMinimum": 0
},
"confidencePct": {
"enum": [
90,
95,
99
],
"type": "number",
"default": 95,
"description": "Confidence level, percent. Two-sided."
},
"resultsPerDay": {
"type": "number",
"description": "Results per day across both versions, split evenly. Gives the number of days.",
"exclusiveMinimum": 0
}
}
}Builds a pass/fail LLM-as-a-judge prompt for one kind of AI step (make, answer, transform, extract, judge, find, predict, plan, act or learn). Returns the prompt with {{input}} and {{result}} placeholders, numbered pass conditions (the block's starter conditions unless you send your own), up to 3 pass and 3 fail examples interleaved, and a JSON or XML answer format.
| Argument | Type | Description |
|---|---|---|
| block* | string | What the AI step does. make: creates new content from a request (an image, a post, slides, advice); answer: replies to a question that has a right answer (a support reply, a policy question, a how-to); transform: reshapes material faithfully (a rewrite, a translation, text to speech); extract: pulls out what's in the input (fields from a PDF, speech to text, text from a photo); judge: gives a label, score or choice that can be checked now (moderation, routing, a guardrail); find: picks the few relevant items out of many (search, recommendations); predict: makes an estimate that can only be checked later (will this lead buy, is this fraud, when will it arrive); plan: turns a goal into steps (a research plan, a multi-step task); act: takes one action with side effects (send an email, take a payment, call an API); learn: updates a model or prompt from examples (a voice clone, prompt tuning). |
| criteria | array | Pass conditions, one per item. Omitted: the starter conditions for the block. |
| outputFormat | string | How the judge answers |
| failedExamples | array | Real results that didn't work; labelled fail in the prompt |
Compares a person's verdicts with an AI judge's verdicts on the same results. Returns the agreement rate, Cohen's kappa with confidence intervals, precision and recall per verdict, the agreement of a judge that always gives the most common verdict, a trust verdict (kappa 0.6 or more with at least 80% recall on every verdict; 0.4 to 0.6 is borderline; under 30 results is too few) and the rows where the two disagree.
| Argument | Type | Description |
|---|---|---|
| csv | string | The same data as CSV or TSV text instead of rows: columns named human and judge (and optionally note), or the first two columns in that order. |
| rows | array | One row per result. Common words are mapped onto two verdicts: worked (worked, pass, yes, true, 1, good, kept, accepted, ok) and didn't work (didn't work, fail, no, false, 0, bad, redo, retry, rejected, wrong). Other labels are compared as written. |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"csv": {
"type": "string",
"maxLength": 500000,
"description": "The same data as CSV or TSV text instead of rows: columns named human and judge (and optionally note), or the first two columns in that order."
},
"rows": {
"type": "array",
"items": {
"type": "object",
"required": [
"human",
"judge"
],
"properties": {
"note": {
"type": "string",
"maxLength": 300,
"description": "What the result was, to recognise it in the disagreements"
},
"human": {
"type": "string",
"maxLength": 100,
"minLength": 1,
"description": "The person's verdict on one result"
},
"judge": {
"type": "string",
"maxLength": 100,
"minLength": 1,
"description": "The AI judge's verdict on the same result"
}
}
},
"maxItems": 5000,
"description": "One row per result. Common words are mapped onto two verdicts: worked (worked, pass, yes, true, 1, good, kept, accepted, ok) and didn't work (didn't work, fail, no, false, 0, bad, redo, retry, rejected, wrong). Other labels are compared as written."
}
}
}The 20 AI-specific launch checks for an AI app (spending caps, rate limits, server-side keys, credit charging, version tags, records of which results worked first time, real cost per result, switching models safely). Returns the checks not yet done, each with why it matters and a request for a coding agent, plus one combined request.
| Argument | Type | Description |
|---|---|---|
| done | array | Ids of checks already done. Ids: spend-cap, rate-limits, server-keys, capped-retries, provider-budget, every-call-charged, deduct-server-side, refund-failures, one-charge-per-request, output-metadata, versioned-regenerations, outcome-events, events-by-version, step-costs, wasted-spend, who-it-fails-for, prompts-in-git, switch-flags, replay-set, shutdown-plan. |
| group | string | Only checks in one group: cost (cost and abuse), charging (charging for ai use), traceability (a record of every ai result), signals (what worked, for whom, at what cost), change (switching safely) |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"done": {
"type": "array",
"items": {
"type": "string",
"maxLength": 60
},
"default": [],
"maxItems": 30,
"description": "Ids of checks already done. Ids: spend-cap, rate-limits, server-keys, capped-retries, provider-budget, every-call-charged, deduct-server-side, refund-failures, one-charge-per-request, output-metadata, versioned-regenerations, outcome-events, events-by-version, step-costs, wasted-spend, who-it-fails-for, prompts-in-git, switch-flags, replay-set, shutdown-plan."
},
"group": {
"enum": [
"cost",
"charging",
"traceability",
"signals",
"change"
],
"type": "string",
"description": "Only checks in one group: cost (cost and abuse), charging (charging for ai use), traceability (a record of every ai result), signals (what worked, for whom, at what cost), change (switching safely)"
}
}
}Looks up list prices for LLMs (USD per million input, output and cached input tokens, and context size), image models (USD per image by quality and size) and video models (USD per second by resolution), with each price's source page, the date prices were checked and any announced API shutdown.
| Argument | Type | Description |
|---|---|---|
| kind | string | llm for text models priced per token, image per image, video per second |
| limit | integer | Most models to return |
| query | string | A model id such as 'gpt-4o' (only exact id matches, if any), or words that must all appear in the id, name or provider, such as 'claude sonnet'. Omitted: every model. |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"kind": {
"enum": [
"llm",
"image",
"video",
"all"
],
"type": "string",
"default": "all",
"description": "llm for text models priced per token, image per image, video per second"
},
"limit": {
"type": "integer",
"default": 25,
"maximum": 100,
"minimum": 1,
"description": "Most models to return"
},
"query": {
"type": "string",
"maxLength": 100,
"description": "A model id such as 'gpt-4o' (only exact id matches, if any), or words that must all appear in the id, name or provider, such as 'claude sonnet'. Omitted: every model."
}
}
}Lists announced API shutdown dates for AI models from OpenAI, Anthropic, Google and other providers, with days left, the suggested replacement, the source page, and links to a cost comparison and a side-by-side test of the replacement where this site has them.
| Argument | Type | Description |
|---|---|---|
| query | string | A model id such as 'gpt-4o' (only exact id matches, if any), or words that must all appear in the model ids or provider, such as 'anthropic'. Omitted: every model. |
| today | string | Date to count days from, YYYY-MM-DD. Omitted: today in UTC. |
| includePast | boolean | Include shutdowns whose date has passed |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"query": {
"type": "string",
"maxLength": 100,
"description": "A model id such as 'gpt-4o' (only exact id matches, if any), or words that must all appear in the model ids or provider, such as 'anthropic'. Omitted: every model."
},
"today": {
"type": "string",
"pattern": "^\\d{4}-\\d{2}-\\d{2}$",
"description": "Date to count days from, YYYY-MM-DD. Omitted: today in UTC."
},
"includePast": {
"type": "boolean",
"default": false,
"description": "Include shutdowns whose date has passed"
}
}
}Compares two priced LLMs on six common task shapes (support answer, invoice extraction, lead scoring, email draft, translation, search re-ranking). Returns list prices, cost per 1,000 requests on each model, the cost ratio, the break-even (how often the new model's results must work first time, relative to the current one, to cost no more per result that works) and the current model's announced shutdown.
| Argument | Type | Description |
|---|---|---|
| to* | string | The new model: an LLM id from /models.json (openai/gpt-6-luna), an OpenRouter id or the model's name |
| from* | string | The current model: an LLM id from /models.json (openai/gpt-4o), an OpenRouter id or the model's name |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"from",
"to"
],
"properties": {
"to": {
"type": "string",
"maxLength": 100,
"minLength": 1,
"description": "The new model: an LLM id from /models.json (openai/gpt-6-luna), an OpenRouter id or the model's name"
},
"from": {
"type": "string",
"maxLength": 100,
"minLength": 1,
"description": "The current model: an LLM id from /models.json (openai/gpt-4o), an OpenRouter id or the model's name"
}
}
}Prices the AI steps behind one request from current list prices (LLM tokens, images, video seconds or a fixed amount) and returns the cost per request, the real cost per result that worked first time (cost per request divided by the worked-first-time share), the monthly spend wasted on results that didn't work, and credit-pricing numbers: margin per result and the credits per request needed to break even or reach a target margin.
| Argument | Type | Description |
|---|---|---|
| steps* | array | The AI steps behind one request, priced from /models.json. Their costs add up to the cost per request. |
| pricing | object | Credit pricing. Defaults: $10 for 1,000 credits, 1 credit per request, a 2.9% + $0.30 fee and a 70% target margin. |
| requestsPerMonth* | number | Requests per month, try-agains included |
| workedFirstTimePct* | number | Share of results that worked first time: no try again, undo, fix, re-ask or overrule. 1 to 100. |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"steps",
"workedFirstTimePct",
"requestsPerMonth"
],
"properties": {
"steps": {
"type": "array",
"items": {
"oneOf": [
{
"type": "object",
"required": [
"kind",
"model",
"inputTokens",
"outputTokens"
],
"properties": {
"kind": {
"type": "string",
"const": "llm"
},
"note": {
"type": "string",
"maxLength": 120,
"description": "What the step does, for your own reference"
},
"model": {
"type": "string",
"minLength": 1,
"description": "LLM model id from /models.json, for example anthropic/claude-haiku-5.5"
},
"inputTokens": {
"type": "integer",
"maximum": 10000000,
"minimum": 0,
"description": "Input tokens per request"
},
"outputTokens": {
"type": "integer",
"maximum": 10000000,
"minimum": 0,
"description": "Output tokens per request"
},
"cachedInputPct": {
"type": "number",
"default": 0,
"maximum": 100,
"minimum": 0,
"description": "Share of input tokens billed at the provider's cached-input price, 0 to 100"
}
}
},
{
"type": "object",
"required": [
"kind",
"model",
"variant"
],
"properties": {
"kind": {
"type": "string",
"const": "image"
},
"note": {
"type": "string",
"maxLength": 120,
"description": "What the step does, for your own reference"
},
"model": {
"type": "string",
"minLength": 1,
"description": "Image model id from /models.json, for example openai/gpt-image-2"
},
"images": {
"type": "integer",
"default": 1,
"maximum": 100,
"minimum": 1,
"description": "Images per request"
},
"variant": {
"type": "string",
"minLength": 1,
"description": "Quality and size label exactly as listed for the model in /models.json"
}
}
},
{
"type": "object",
"required": [
"kind",
"model",
"variant",
"seconds"
],
"properties": {
"kind": {
"type": "string",
"const": "video"
},
"note": {
"type": "string",
"maxLength": 120,
"description": "What the step does, for your own reference"
},
"model": {
"type": "string",
"minLength": 1,
"description": "Video model id from /models.json, for example google/veo-3.1-fast"
},
"seconds": {
"type": "number",
"maximum": 600,
"description": "Seconds of video per request",
"exclusiveMinimum": 0
},
"variant": {
"type": "string",
"minLength": 1,
"description": "Resolution and audio label exactly as listed for the model in /models.json"
}
}
},
{
"type": "object",
"required": [
"kind",
"usd"
],
"properties": {
"usd": {
"type": "number",
"maximum": 1000,
"minimum": 0,
"description": "What this step costs per request, in USD"
},
"kind": {
"type": "string",
"const": "custom"
},
"label": {
"type": "string",
"maxLength": 120,
"description": "What the step is, for example a speech-to-text API"
}
}
}
]
},
"maxItems": 4,
"minItems": 1,
"description": "The AI steps behind one request, priced from /models.json. Their costs add up to the cost per request."
},
"pricing": {
"type": "object",
"default": {},
"properties": {
"feePct": {
"type": "number",
"default": 2.9,
"minimum": 0,
"description": "Payment processor fee, percent of the pack price",
"exclusiveMaximum": 100
},
"feeFixedUsd": {
"type": "number",
"default": 0.3,
"minimum": 0,
"description": "Payment processor fixed fee per pack, USD"
},
"packCredits": {
"type": "number",
"default": 1000,
"description": "Credits in that pack",
"exclusiveMinimum": 0
},
"packPriceUsd": {
"type": "number",
"default": 10,
"description": "What a user pays for one credit pack, USD",
"exclusiveMinimum": 0
},
"targetMarginPct": {
"type": "number",
"default": 70,
"minimum": 0,
"description": "Gross margin wanted on each result that worked, percent",
"exclusiveMaximum": 100
},
"chargeOnlyWorked": {
"type": "boolean",
"default": false,
"description": "True if users are charged only for results that worked"
},
"creditsPerRequest": {
"type": "number",
"default": 1,
"description": "Credits charged each time the AI runs",
"exclusiveMinimum": 0
}
},
"description": "Credit pricing. Defaults: $10 for 1,000 credits, 1 credit per request, a 2.9% + $0.30 fee and a 70% target margin."
},
"requestsPerMonth": {
"type": "number",
"maximum": 1000000000,
"minimum": 0,
"description": "Requests per month, try-agains included"
},
"workedFirstTimePct": {
"type": "number",
"maximum": 100,
"minimum": 1,
"description": "Share of results that worked first time: no try again, undo, fix, re-ask or overrule. 1 to 100."
}
}
}We read every tool's name, description and inputs the way the model will, looking for instructions hidden from you. A clean result means no warning signs, not a guarantee. Scanned Nov 12, 2026.
| workedExamples | array | Real results that worked first time; labelled pass in the prompt |
| includeReasoning | boolean | Whether the judge writes a short reason before its verdict |
| featureDescription | string | Who uses the feature, what they ask for and what a good result gives them |
{
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"block"
],
"properties": {
"block": {
"enum": [
"make",
"answer",
"transform",
"extract",
"judge",
"find",
"predict",
"plan",
"act",
"learn"
],
"type": "string",
"description": "What the AI step does. make: creates new content from a request (an image, a post, slides, advice); answer: replies to a question that has a right answer (a support reply, a policy question, a how-to); transform: reshapes material faithfully (a rewrite, a translation, text to speech); extract: pulls out what's in the input (fields from a PDF, speech to text, text from a photo); judge: gives a label, score or choice that can be checked now (moderation, routing, a guardrail); find: picks the few relevant items out of many (search, recommendations); predict: makes an estimate that can only be checked later (will this lead buy, is this fraud, when will it arrive); plan: turns a goal into steps (a research plan, a multi-step task); act: takes one action with side effects (send an email, take a payment, call an API); learn: updates a model or prompt from examples (a voice clone, prompt tuning)."
},
"criteria": {
"type": "array",
"items": {
"type": "string",
"maxLength": 500,
"minLength": 1
},
"maxItems": 10,
"minItems": 1,
"description": "Pass conditions, one per item. Omitted: the starter conditions for the block."
},
"outputFormat": {
"enum": [
"json",
"xml-tags"
],
"type": "string",
"default": "json",
"description": "How the judge answers"
},
"failedExamples": {
"type": "array",
"items": {
"type": "object",
"required": [
"result"
],
"properties": {
"input": {
"type": "string",
"default": "",
"maxLength": 4000,
"description": "What the step received"
},
"result": {
"type": "string",
"maxLength": 4000,
"description": "What the step produced, in full"
}
}
},
"default": [],
"maxItems": 3,
"description": "Real results that didn't work; labelled fail in the prompt"
},
"workedExamples": {
"type": "array",
"items": {
"type": "object",
"required": [
"result"
],
"properties": {
"input": {
"type": "string",
"default": "",
"maxLength": 4000,
"description": "What the step received"
},
"result": {
"type": "string",
"maxLength": 4000,
"description": "What the step produced, in full"
}
}
},
"default": [],
"maxItems": 3,
"description": "Real results that worked first time; labelled pass in the prompt"
},
"includeReasoning": {
"type": "boolean",
"default": true,
"description": "Whether the judge writes a short reason before its verdict"
},
"featureDescription": {
"type": "string",
"default": "",
"maxLength": 2000,
"description": "Who uses the feature, what they ask for and what a good result gives them"
}
}
}