compare_models

Compare several models side by side at a given volume: cost per request and per month, prices, context window, output limit, release date and knowledge cutoff, from AI Gateway's live catalog. Pass ids like 'openai/gpt-4.1-mini', or a provider name ('anthropic', 'openai', 'google') for its newest models, so you don't need to know which models are current.

read-only

Argument	Type	Description
models*	array	Model ids like 'openai/gpt-4.1-mini', or provider names like 'anthropic' for that provider's newest models
cached_share	number	Share of input tokens served from the prompt cache, 0 to 1
input_tokens*	integer	Input tokens per request: system prompt, retrieved documents, history and the user's message
output_tokens*	integer	Output tokens per request: the length of a typical reply
requests_per_day*	integer	Expected requests per day, for the monthly cost

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "required": [
    "models",
    "input_tokens",
    "output_tokens",
    "requests_per_day"
  ],
  "properties": {
    "models": {
      "type": "array",
      "items": {
        "type": "string",
        "maxLength": 100,
        "minLength": 2
      },
      "maxItems": 12,
      "minItems": 1,
      "description": "Model ids like 'openai/gpt-4.1-mini', or provider names like 'anthropic' for that provider's newest models"
    },
    "cached_share": {
      "type": "number",
      "maximum": 1,
      "minimum": 0,
      "description": "Share of input tokens served from the prompt cache, 0 to 1"
    },
    "input_tokens": {
      "type": "integer",
      "maximum": 2000000,
      "minimum": 0,
      "description": "Input tokens per request: system prompt, retrieved documents, history and the user's message"
    },
    "output_tokens": {
      "type": "integer",
      "maximum": 200000,
      "minimum": 0,
      "description": "Output tokens per request: the length of a typical reply"
    },
    "requests_per_day": {
      "type": "integer",
      "maximum": 100000000,
      "minimum": 1,
      "description": "Expected requests per day, for the monthly cost"
    }
  }
}

count_tokens

Count the tokens in a text (an estimate with OpenAI's cl100k tokenizer; other model families differ slightly).

read-only

Argument	Type	Description
text*	string	The text to count, such as a system prompt or a typical user message

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "required": [
    "text"
  ],
  "properties": {
    "text": {
      "type": "string",
      "maxLength": 100000,
      "minLength": 1,
      "description": "The text to count, such as a system prompt or a typical user message"
    }
  }
}

define_term

A plain-language definition of an AI product concept (tokens, temperature, RAG, embeddings, LLM-as-judge, prompt injection…), with the lessons that teach it.

read-only

Argument	Type	Description
term*	string	The AI term to define, e.g. 'RAG', 'temperature' or 'prompt caching'

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "required": [
    "term"
  ],
  "properties": {
    "term": {
      "type": "string",
      "maxLength": 100,
      "minLength": 1,
      "description": "The AI term to define, e.g. 'RAG', 'temperature' or 'prompt caching'"
    }
  }
}

design_eval_suite

Build an evaluation suite for a planned AI feature with AI PM Lab's method: test cases (typical, edge, out of scope, attacks), graders with a judge rubric, pass bars, and a JSON test set. Returns instructions to follow with the user's text.

read-only

Argument	Type	Description
feature	string	What the AI feature does, who uses it, and what it must never do

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "properties": {
    "feature": {
      "type": "string",
      "maxLength": 50000,
      "description": "What the AI feature does, who uses it, and what it must never do"
    }
  }
}

estimate_cost

What a model call costs per request, per day and per month at a given volume, from AI Gateway's live prices. Model ids look like 'openai/gpt-4.1-mini' or 'anthropic/claude-sonnet-4.5'.

read-only

Argument	Type	Description
model*	string	A model id like 'openai/gpt-4.1-mini' or 'anthropic/claude-sonnet-4'
cached_share	number	Share of input tokens served from the prompt cache, 0 to 1
input_tokens*	integer	Input tokens per request: system prompt, retrieved documents, history and the user's message
output_tokens*	integer	Output tokens per request: the length of a typical reply
requests_per_day*	integer	Expected requests per day, for the monthly cost

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "required": [
    "model",
    "input_tokens",
    "output_tokens",
    "requests_per_day"
  ],
  "properties": {
    "model": {
      "type": "string",
      "maxLength": 100,
      "minLength": 3,
      "description": "A model id like 'openai/gpt-4.1-mini' or 'anthropic/claude-sonnet-4'"
    },
    "cached_share": {
      "type": "number",
      "maximum": 1,
      "minimum": 0,
      "description": "Share of input tokens served from the prompt cache, 0 to 1"
    },
    "input_tokens": {
      "type": "integer",
      "maximum": 2000000,
      "minimum": 0,
      "description": "Input tokens per request: system prompt, retrieved documents, history and the user's message"
    },
    "output_tokens": {
      "type": "integer",
      "maximum": 200000,
      "minimum": 0,
      "description": "Output tokens per request: the length of a typical reply"
    },
    "requests_per_day": {
      "type": "integer",
      "maximum": 100000000,
      "minimum": 1,
      "description": "Expected requests per day, for the monthly cost"
    }
  }
}

get_lesson

A lesson in full: the product decision it informs, its key idea and concepts, things to try with what to notice in each (from recorded runs), what a PM takes away and questions to ask engineers, with links to the lesson, its 3D tour and its challenge.

read-only

Argument	Type	Description
slug*	string	The lesson's slug, from search_lessons

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "required": [
    "slug"
  ],
  "properties": {
    "slug": {
      "enum": [
        "how-llms-work",
        "writing-good-prompts",
        "hallucination-and-grounding",
        "embeddings-and-semantic-search",
        "prompting-vs-rag-vs-fine-tuning",
        "prompt-caching-and-cost",
        "structured-outputs",
        "how-ai-agents-work",
        "retrieval-augmented-generation",
        "tool-design-and-mcp",
        "multi-agent-systems",
        "workflows-vs-agents",
        "ai-evals",
        "reasoning-models",
        "production-monitoring",
        "prompt-injection",
        "model-routing",
        "human-in-the-loop"
      ],
      "type": "string",
      "description": "The lesson's slug, from search_lessons"
    }
  }
}

list_challenges

The challenges (games scored from real recorded runs, with leaderboards) and what each asks you to do.

read-only

No arguments.

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "properties": {}
}

list_experiments

The real runs recorded on one of AI PM Lab's 3D pages or lessons, each described by its setup (e.g. which defences or settings were on). Replay one to see what happened.

read-only

Argument	Type	Description
page*	string	A 3D page (injection, rag, evals, agent-loop, next-token, embeddings) or a lesson's slug (e.g. prompt-injection, model-routing, hallucination-and-grounding, human-in-the-loop)

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "required": [
    "page"
  ],
  "properties": {
    "page": {
      "enum": [
        "injection",
        "rag",
        "evals",
        "agent-loop",
        "next-token",
        "embeddings",
        "how-ai-agents-work",
        "how-llms-work",
        "retrieval-augmented-generation",
        "embeddings-and-semantic-search",
        "prompt-caching-and-cost",
        "multi-agent-systems",
        "prompt-injection",
        "ai-evals",
        "prompting-vs-rag-vs-fine-tuning",
        "human-in-the-loop",
        "hallucination-and-grounding",
        "production-monitoring",
        "writing-good-prompts",
        "reasoning-models",
        "model-routing",
        "structured-outputs",
        "tool-design-and-mcp",
        "workflows-vs-agents"
      ],
      "type": "string",
      "description": "A 3D page (injection, rag, evals, agent-loop, next-token, embeddings) or a lesson's slug (e.g. prompt-injection, model-routing, hallucination-and-grounding, human-in-the-loop)"
    }
  }
}

plan_ai_launch

A phased roadmap from where an AI feature is today to launch and its first month live: frame, prototype and model, evals, hardening, pilot, launch, monitoring. Each phase has exit criteria and the tests and checks for this feature, backed by findings from AI PM Lab's recorded real runs (injection, grounding, evals, agent cost). Returns instructions to follow with the user's text.

read-only

Argument	Type	Description
feature	string	What the AI feature does, who uses it, what it must never do, and where the work stands today

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "properties": {
    "feature": {
      "type": "string",
      "maxLength": 50000,
      "description": "What the AI feature does, who uses it, what it must never do, and where the work stands today"
    }
  }
}

prep_planning_meeting

The questions a PM should ask engineers about a planned AI feature, with why each matters and what a good answer sounds like. Returns instructions to follow with the user's text.

read-only

Argument	Type	Description
feature	string	A few sentences about the AI feature

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "properties": {
    "feature": {
      "type": "string",
      "maxLength": 50000,
      "description": "A few sentences about the AI feature"
    }
  }
}

price_ai_feature

Work out an AI feature's unit economics and pricing: cost per request and per user (light, typical, heavy), levers like caching and routing, pricing options with margins, guardrails, and the assumptions to validate. Returns instructions to follow with the user's text.

read-only

Argument	Type	Description
feature	string	What the feature does, the model it uses, expected usage, and how the product is priced today

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "properties": {
    "feature": {
      "type": "string",
      "maxLength": 50000,
      "description": "What the feature does, the model it uses, expected usage, and how the product is priced today"
    }
  }
}

quiz_me

A round from AI PM Lab's challenges, built from real recorded runs: Call It (predict what a model did), Spot the Failure (ship or flag a real output) or Which Wins? (pick the better of two setups). Returns the question without its answer; call again with the round id and the user's choice to mark it, see how players on the site called it, and get the next round.

read-only

Argument	Type	Description
game	string	call-it, spot-the-failure or which-wins; any if left out
round	string	The id of the round the user is answering, from the last quiz_me result
answer	string	The user's choice for that round: its letter

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "properties": {
    "game": {
      "enum": [
        "call-it",
        "spot-the-failure",
        "which-wins"
      ],
      "type": "string",
      "description": "call-it, spot-the-failure or which-wins; any if left out"
    },
    "round": {
      "type": "string",
      "maxLength": 200,
      "description": "The id of the round the user is answering, from the last quiz_me result"
    },
    "answer": {
      "type": "string",
      "maxLength": 200,
      "description": "The user's choice for that round: its letter"
    }
  }
}

replay_experiment

What happened in one recorded real run (e.g. whether a prompt injection leaked data with given defences, or how many answers were made up without a facts sheet): the setup, each step, the result and the model's reply, with a link to run it on the site.

read-only

Argument	Type	Description
run*	integer	The run's number from list_experiments
page*	string	A 3D page (injection, rag, evals, agent-loop, next-token, embeddings) or a lesson's slug (e.g. prompt-injection, model-routing, hallucination-and-grounding, human-in-the-loop)

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "required": [
    "page",
    "run"
  ],
  "properties": {
    "run": {
      "type": "integer",
      "maximum": 500,
      "minimum": 1,
      "description": "The run's number from list_experiments"
    },
    "page": {
      "enum": [
        "injection",
        "rag",
        "evals",
        "agent-loop",
        "next-token",
        "embeddings",
        "how-ai-agents-work",
        "how-llms-work",
        "retrieval-augmented-generation",
        "embeddings-and-semantic-search",
        "prompt-caching-and-cost",
        "multi-agent-systems",
        "prompt-injection",
        "ai-evals",
        "prompting-vs-rag-vs-fine-tuning",
        "human-in-the-loop",
        "hallucination-and-grounding",
        "production-monitoring",
        "writing-good-prompts",
        "reasoning-models",
        "model-routing",
        "structured-outputs",
        "tool-design-and-mcp",
        "workflows-vs-agents"
      ],
      "type": "string",
      "description": "A 3D page (injection, rag, evals, agent-loop, next-token, embeddings) or a lesson's slug (e.g. prompt-injection, model-routing, hallucination-and-grounding, human-in-the-loop)"
    }
  }
}

review_ai_incident

A blameless post-incident review for an AI feature: where it failed (input, retrieval, prompt, model, tools, output checks), why evals and monitoring missed it, actions, and the new eval cases that would catch it next time. Returns instructions to follow with the user's text.

read-only

Argument	Type	Description
incident	string	What the feature does, what went wrong, and what's known so far

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "properties": {
    "incident": {
      "type": "string",
      "maxLength": 50000,
      "description": "What the feature does, what went wrong, and what's known so far"
    }
  }
}

review_ai_spec

Check an AI feature spec or PRD against what AI PM Lab teaches: approach, model and cost, prompt, grounding, output contract, tools, security, human oversight, evals, monitoring, fallbacks. Gaps link to lessons. Returns instructions to follow with the user's text.

read-only

Argument	Type	Description
spec	string	The spec or PRD text

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "properties": {
    "spec": {
      "type": "string",
      "maxLength": 50000,
      "description": "The spec or PRD text"
    }
  }
}

review_launch_readiness

A go/no-go review of an AI feature's launch plan: evals, grounding, attacks, approvals, cost, latency, monitoring, rollout and support, each marked ready, at risk or missing, with blockers and a launch-day watch list. Returns instructions to follow with the user's text.

read-only

Argument	Type	Description
plan	string	The feature, what was tested, and the launch plan

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "properties": {
    "plan": {
      "type": "string",
      "maxLength": 50000,
      "description": "The feature, what was tested, and the launch plan"
    }
  }
}

search_lessons

Find AI PM Lab lessons on a topic (e.g. RAG, evals, agents, prompt injection, cost). Returns the best matches with their key idea and link.

read-only

Argument	Type	Description
query*	string	A topic or question, e.g. 'how do I stop hallucinations'

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "required": [
    "query"
  ],
  "properties": {
    "query": {
      "type": "string",
      "maxLength": 200,
      "minLength": 1,
      "description": "A topic or question, e.g. 'how do I stop hallucinations'"
    }
  }
}

write_acceptance_criteria

Acceptance criteria for an AI feature or user story that can actually be tested: Given/When/Then with measurable pass bars on a test set, must-pass safety criteria, latency and cost budgets, and example test cases, ready to paste into a ticket. Returns instructions to follow with the user's text.

read-only

Argument	Type	Description
feature	string	The AI feature or user story, who uses it, and what it must never do

Input schema (JSON)

{
  "type": "object",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "properties": {
    "feature": {
      "type": "string",
      "maxLength": 50000,
      "description": "The AI feature or user story, who uses it, and what it must never do"
    }
  }
}

AI PM Lab

Try it

compare_models

AI PM Lab

Try it

compare_models

Tools (18)