Servers Playground How we check Add a server Log in

IA-QA Eval & RAG — 38 tools

Working

io.github.JcJamet/ia-qa-evalv1.0.1 GitHub verified

LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key.

AI & Agents Developer Tools

Source Registry entry

Maintain this server? Claim it

Try it

Call this server's tools from your browser. Nothing to install.

https://www.ia-qa.com/mcp/eval

Requests go through mcptry because browsers can't call most MCP servers directly. Header values are sent to the server for that request only and are never stored or logged.

Tool list from our last check. Connect live to refresh it.

ab_test_report Read-only

Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns statistical comparison: mean, median, std deviation, winner, and improvement percentage.

variant_aobject, required

First variant configuration with name and score array

variant_bobject, required

Second variant configuration with name and score array

IA-QA Eval & RAG — 38 tools MCP server | mcptry

Tools (38)

ab_test_report

read-only

Argument	Type	Description
variant_a*	object	First variant configuration with name and score array
variant_b*	object	Second variant configuration with name and score array

Input schema (JSON)

{
  "type": "object",
  "required": [
    "variant_a",
    "variant_b"
  ],
  "properties": {
    "variant_a": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string",
          "description": "Name/label for variant A"
        },
        "scores": {
          "type": "array",
          "items": {
            "type": "number"
          },
          "description": "Array of scores (0-100)"
        }
      },
      "description": "First variant configuration with name and score array"
    },
    "variant_b": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string",
          "description": "Name/label for variant B"
        },
        "scores": {
          "type": "array",
          "items": {
            "type": "number"
          },
          "description": "Array of scores (0-100)"
        }
      },
      "description": "Second variant configuration with name and score array"
    }
  }
}

analyze_responses

Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a `reference` (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). For a 2-way head-to-head with structural JSON diff use compare_responses instead.

read-only

Argument	Type	Description
reference	string	Optional ground-truth answer. If set, each output is also ranked by closeness to it and the closest one is named.
responses*	array	The outputs to analyze (same task, N models/prompts/versions). Each item is a plain string or { "label": "GPT-4o", "text": "..." }. At least 2 required.

Input schema (JSON)

{
  "type": "object",
  "required": [
    "responses"
  ],
  "properties": {
    "reference": {
      "type": "string",
      "description": "Optional ground-truth answer. If set, each output is also ranked by closeness to it and the closest one is named."
    },
    "responses": {
      "type": "array",
      "items": {
        "type": "object",
        "required": [
          "text"
        ],
        "properties": {
          "text": {
            "type": "string",
            "description": "The produced output"
          },
          "label": {
            "type": "string",
            "description": "Human name for this candidate (e.g. model id)"
          }
        }
      },
      "minItems": 2,
      "description": "The outputs to analyze (same task, N models/prompts/versions). Each item is a plain string or { \"label\": \"GPT-4o\", \"text\": \"...\" }. At least 2 required."
    }
  }
}

bm25_score

Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.

read-only

Argument	Type	Description
b	number	Length normalization factor (default: 0.75)
k1	number	Term frequency saturation (default: 1.5)
query*	string	The search query
top_k	number	Return top K results (default: all)
documents*	array	Array of documents to rank

Input schema (JSON)

{
  "type": "object",
  "required": [
    "query",
    "documents"
  ],
  "properties": {
    "b": {
      "type": "number",
      "description": "Length normalization factor (default: 0.75)"
    },
    "k1": {
      "type": "number",
      "description": "Term frequency saturation (default: 1.5)"
    },
    "query": {
      "type": "string",
      "description": "The search query"
    },
    "top_k": {
      "type": "number",
      "description": "Return top K results (default: all)"
    },
    "documents": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Array of documents to rank"
    }
  }
}

build_rag_prompt

Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.

read-only

Argument	Type	Description
query*	string	The user question to answer
chunks*	array	Retrieved context chunks with .text (required), .source (optional), .score (optional)
language	string	Response language instruction (e.g. "French", "Spanish")
cite_sources	boolean	Add [1], [2] citation numbers (default: true)
max_context_tokens	number	Max tokens for context section (default: 2000)
system_instruction	string	Custom system instruction (default: standard RAG grounding instruction)

Input schema (JSON)

{
  "type": "object",
  "required": [
    "query",
    "chunks"
  ],
  "properties": {
    "query": {
      "type": "string",
      "description": "The user question to answer"
    },
    "chunks": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "text": {
            "type": "string"
          },
          "score": {
            "type": "number"
          },
          "source": {
            "type": "string"
          }
        }
      },
      "description": "Retrieved context chunks with .text (required), .source (optional), .score (optional)"
    },
    "language": {
      "type": "string",
      "description": "Response language instruction (e.g. \"French\", \"Spanish\")"
    },
    "cite_sources": {
      "type": "boolean",
      "description": "Add [1], [2] citation numbers (default: true)"
    },
    "max_context_tokens": {
      "type": "number",
      "description": "Max tokens for context section (default: 2000)"
    },
    "system_instruction": {
      "type": "string",
      "description": "Custom system instruction (default: standard RAG grounding instruction)"
    }
  }
}

compare_models

Compare 2-5 AI models side by side: context window, pricing, multimodal, reasoning capabilities, and provider. Returns a comparison table with a recommendation based on your use case.

read-only

Argument	Type	Description
models*	array	Array of 2-5 model names (e.g. ["gpt-4o","claude-3.5-sonnet","gemini-3.5-flash"])
use_case	string	Optimize recommendation for this criterion

Input schema (JSON)

{
  "type": "object",
  "required": [
    "models"
  ],
  "properties": {
    "models": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Array of 2-5 model names (e.g. [\"gpt-4o\",\"claude-3.5-sonnet\",\"gemini-3.5-flash\"])"
    },
    "use_case": {
      "enum": [
        "cost",
        "context",
        "reasoning",
        "multimodal",
        "speed"
      ],
      "type": "string",
      "description": "Optimize recommendation for this criterion"
    }
  }
}

compare_responses

Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it and picks the closer one. If `model` + `api_key` are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.

read-only

Argument	Type	Description
task	string	The task/prompt both outputs were answering — used by the LLM judge for context
model	string	Optional judge model id (BYOK). When set with api_key, an LLM judge picks a qualitative winner.
api_key	string	Optional API key for the judge model (BYOK). Used only for the judge call; never stored.
label_a	string	Label for output A (e.g. "GPT-4o", "v1.0")
label_b	string	Label for output B (e.g. "GPT-5-nano", "v1.1")
reference	string	Optional ground-truth / expected answer. If set, each output is scored against it and the closer one wins (deterministic).
check_json	boolean	Try to parse as JSON and compare structurally (keys, types, values)

consistency_check

Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. For true semantic consistency, use run_semantic_tests with embedding mode. Essential for determinism testing.

read-only

Argument	Type	Description
responses*	array	Array of 2+ LLM responses to compare (same prompt, different runs)
check_facts	boolean	Check for contradictory numbers/facts across responses (default: true)

Input schema (JSON)

{
  "type": "object",
  "required": [
    "responses"
  ],
  "properties": {
    "responses": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Array of 2+ LLM responses to compare (same prompt, different runs)"
    },
    "check_facts": {
      "type": "boolean",
      "description": "Check for contradictory numbers/facts across responses (default: true)"
    }
  }
}

context_window_check

Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.

read-only

Argument	Type	Description
model*	string	Target model name (e.g. gpt-4o, claude-3.5-sonnet)
messages*	array	Array of messages (system/user/assistant)
max_output_tokens	number	Reserved tokens for output (default: 4096)

Input schema (JSON)

{
  "type": "object",
  "required": [
    "messages",
    "model"
  ],
  "properties": {
    "model": {
      "type": "string",
      "description": "Target model name (e.g. gpt-4o, claude-3.5-sonnet)"
    },
    "messages": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "role": {
            "type": "string"
          },
          "content": {
            "type": "string"
          }
        }
      },
      "description": "Array of messages (system/user/assistant)"
    },
    "max_output_tokens": {
      "type": "number",
      "description": "Reserved tokens for output (default: 4096)"
    }
  }
}

conversation_analyze

Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.

read-only

Argument	Type	Description
messages*	array	Conversation messages in order

Input schema (JSON)

{
  "type": "object",
  "required": [
    "messages"
  ],
  "properties": {
    "messages": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "role": {
            "enum": [
              "system",
              "user",
              "assistant"
            ],
            "type": "string"
          },
          "content": {
            "type": "string"
          }
        }
      },
      "description": "Conversation messages in order"
    }
  }
}

cot_analyzer

Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.

read-only

Argument	Type	Description
reasoning*	string	The CoT / reasoning trace text (e.g. from <think> tags or step-by-step output)
expected_conclusion	string	Expected final answer to check against (optional)

Input schema (JSON)

{
  "type": "object",
  "required": [
    "reasoning"
  ],
  "properties": {
    "reasoning": {
      "type": "string",
      "description": "The CoT / reasoning trace text (e.g. from <think> tags or step-by-step output)"
    },
    "expected_conclusion": {
      "type": "string",
      "description": "Expected final answer to check against (optional)"
    }
  }
}

count_tokens

Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Call this BEFORE sending any text to an LLM API to check if it fits within the model context window and to estimate cost. Returns token estimate, character count, and word count.

read-only

Argument	Type	Description
input*	string	Text to count tokens for

Input schema (JSON)

{
  "type": "object",
  "required": [
    "input"
  ],
  "properties": {
    "input": {
      "type": "string",
      "description": "Text to count tokens for"
    }
  }
}

embedding_similarity

Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: for true semantic similarity with vector embeddings, use run_semantic_tests with mode="embeddings" and your OpenAI API key. Supports single pair or batch mode with pipe-separated pairs. Useful for RAG retrieval testing, semantic search evaluation, and text deduplication.

read-only

Argument	Type	Description
batch	array	Batch mode: array of { text_a, text_b } pairs. Overrides text_a/text_b if provided.
text_a	string	First text to compare (single-pair mode)
text_b	string	Second text to compare (single-pair mode)
methods	array	Algorithms to use (default: all three). Options: "bow", "tfidf", "ngram"

Input schema (JSON)

{
  "type": "object",
  "properties": {
    "batch": {
      "type": "array",
      "items": {
        "type": "object",
        "required": [
          "text_a",
          "text_b"
        ],
        "properties": {
          "text_a": {
            "type": "string"
          },
          "text_b": {
            "type": "string"
          }
        }
      },
      "description": "Batch mode: array of { text_a, text_b } pairs. Overrides text_a/text_b if provided."
    },
    "text_a": {
      "type": "string",
      "description": "First text to compare (single-pair mode)"
    },
    "text_b": {
      "type": "string",
      "description": "Second text to compare (single-pair mode)"
    },
    "methods": {
      "type": "array",
      "items": {
        "enum": [
          "bow",
          "tfidf",
          "ngram"
        ],
        "type": "string"
      },
      "description": "Algorithms to use (default: all three). Options: \"bow\", \"tfidf\", \"ngram\""
    }
  }
}

estimate_llm_cost

Estimate the API cost in USD for a given model and token counts. Supports all major 2024–2026 models: GPT-4o, GPT-4.1, o3, o4-mini, Claude Opus 4, Claude Sonnet 4/4.5, Gemini 2.5 Pro/Flash, DeepSeek V3/R1, Grok 3, and legacy models.

read-only

Argument	Type	Description
model*	string	Model name, e.g. "gpt-4o", "claude-3.5-sonnet", "deepseek-v3"
input_tokens*	number	Number of input/prompt tokens
output_tokens	number	Number of output/completion tokens (default: 0)

Input schema (JSON)

{
  "type": "object",
  "required": [
    "model",
    "input_tokens"
  ],
  "properties": {
    "model": {
      "type": "string",
      "description": "Model name, e.g. \"gpt-4o\", \"claude-3.5-sonnet\", \"deepseek-v3\""
    },
    "input_tokens": {
      "type": "number",
      "description": "Number of input/prompt tokens"
    },
    "output_tokens": {
      "type": "number",
      "description": "Number of output/completion tokens (default: 0)"
    }
  }
}

extract_json_from_text

Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON. Call this whenever an LLM returns structured data mixed with explanation text instead of raw JSON.

read-only

Argument	Type	Description
input*	string	Raw text (e.g., LLM output) that may contain a JSON object or array

Input schema (JSON)

{
  "type": "object",
  "required": [
    "input"
  ],
  "properties": {
    "input": {
      "type": "string",
      "description": "Raw text (e.g., LLM output) that may contain a JSON object or array"
    }
  }
}

find_tool

Search available MCP tools by keyword or category before calling them. Returns matching tool names, descriptions, and optionally their inputSchemas. Call this when you are unsure which tool to use or want to explore the catalogue. Categories: data, encoding, text, llm, qa, rag, dev, security, web.

read-only

Argument	Type	Description
query*	string	Keyword(s) to search in tool name and description (e.g. "cors", "token", "vector", "json")
category	string	Optional: filter by category — data \| encoding \| text \| llm \| qa \| rag \| dev \| security \| web
max_results	number	Maximum tools to return (default 10, max 50). Results are ranked by IDF-weighted relevance, so common words like "test" do not inflate the list.
with_schema	boolean	Set true to include inputSchema in results (default: false)

Input schema (JSON)

{
  "type": "object",
  "required": [
    "query"
  ],
  "properties": {
    "query": {
      "type": "string",
      "description": "Keyword(s) to search in tool name and description (e.g. \"cors\", \"token\", \"vector\", \"json\")"
    },
    "category": {
      "type": "string",
      "description": "Optional: filter by category — data | encoding | text | llm | qa | rag | dev | security | web"
    },
    "max_results": {
      "type": "number",
      "description": "Maximum tools to return (default 10, max 50). Results are ranked by IDF-weighted relevance, so common words like \"test\" do not inflate the list."
    },
    "with_schema": {
      "type": "boolean",
      "description": "Set true to include inputSchema in results (default: false)"
    }
  }
}

generate_ci_workflow

Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA. Two gate types, combinable: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (LLM quality gate, needs a provider API key as a repo secret), and "cli_checks" runs deterministic primitives via npx @ia-qa/cli (secret scan, prompt-injection scan, security headers…) whose exit code fails the build. Deterministic template — no LLM call, no API key, same inputs give the same file. Returns the YAML, the secrets to create, and the remaining steps. Pair with generate_eval_yaml to produce the contract itself.

read-only

Argument	Type	Description
cron	string	Cron expression when triggers include 'schedule' (default: '0 6 * * 1' — Mondays 06:00 UTC).
gate	string	Which gate to emit. eval_contract = LLM eval via the action (default). cli_checks = deterministic CLI assertions. selector_drift = an E2E selector-drift gate via @ia-qa/self-healing (boots the app, captures, diffs against the committed baseline, branches on exit code 0/1/2). both = CLI checks + eval. all = CLI checks, then drift, then eval.
provider	string	LLM provider the contract runs against — decides which repository secret the workflow wires (default: groq).
triggers	array	Workflow triggers (default: push + pull_request).
cli_tools	array	IA-QA tool names to run as deterministic gates, e.g. ["secret_scan","prompt_injection_scan"]. Tools with no known CI recipe get a --stdin step flagged in notes.

generate_eval_yaml

Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.

read-only

Argument	Type	Description
task_type	string	Optional task type hint to guide evaluator selection.
description*	string	Plain-language description of what the LLM under test should do. Be specific: describe inputs, expected behaviour, and constraints.
system_prompt	string	Optional system prompt of the LLM under test. Helps generate more accurate test cases.
scenario_count	number	Number of scenarios to generate (default: 5). Covers happy path + edge cases + adversarial.

Input schema (JSON)

{
  "type": "object",
  "required": [
    "description"
  ],
  "properties": {
    "task_type": {
      "enum": [
        "rag",
        "summarization",
        "classification",
        "safety",
        "customer_support",
        "code_gen"
      ],
      "type": "string",
      "description": "Optional task type hint to guide evaluator selection."
    },
    "description": {
      "type": "string",
      "description": "Plain-language description of what the LLM under test should do. Be specific: describe inputs, expected behaviour, and constraints."
    },
    "system_prompt": {
      "type": "string",
      "description": "Optional system prompt of the LLM under test. Helps generate more accurate test cases."
    },
    "scenario_count": {
      "enum": [
        3,
        5,
        8
      ],
      "type": "number",
      "description": "Number of scenarios to generate (default: 5). Covers happy path + edge cases + adversarial."
    }
  }
}

get_testing_guidelines

Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset, selector-drift, qa-recipes, playbooks. Not sure where to start testing an LLM, RAG pipeline or agent? Call without a topic (or with "start-here"): it maps what you are testing to the tools to call and the output field to gate CI on. A plain question such as "how do I test my RAG" also resolves to the right topic.

read-only

Argument	Type	Description
topic	string	The testing topic to retrieve guidelines for. Omit to get the start-here map and the full list of available topics.

Input schema (JSON)

{
  "type": "object",
  "properties": {
    "topic": {
      "enum": [
        "start-here",
        "llm-unit-testing",
        "rag-pipeline",
        "prompt-stability",
        "prompt-ab-testing",
        "embedding-quality",
        "eval-framework",
        "semantic-testing",
        "auto-testing",
        "security",
        "api-testing",
        "ci-cd",
        "multimodal",
        "llm-data-security",
        "agent-observability",
        "pro-tips",
        "learning-paths",
        "golden-dataset",
        "selector-drift",
        "qa-recipes",
        "playbooks"
      ],
      "type": "string",
      "description": "The testing topic to retrieve guidelines for. Omit to get the start-here map and the full list of available topics."
    }
  }
}

hallucination_check

Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its best-matching source sentence, so a number only counts as support when it sits on the SAME statement ("founded in 1998" is not grounded by "sold 1998 units"), and a negation or antonym flip against that sentence returns verdict "contradicted" — the corrupted-fact hallucination that reuses source vocabulary. Limitations: still lexical — it cannot follow a paraphrase, a synonym, or multi-sentence reasoning, so a "well_grounded" verdict means "nothing lexical found", never "verified true". For entailment use run_semantic_tests (NLI/embedding) or a calibrated judge.

read-only

Argument	Type	Description
answer*	string	The LLM-generated answer to verify
strict	boolean	If true, every sentence in the answer must be supported (default: false)
context*	string	The source/reference text that should ground the answer

Input schema (JSON)

{
  "type": "object",
  "required": [
    "answer",
    "context"
  ],
  "properties": {
    "answer": {
      "type": "string",
      "description": "The LLM-generated answer to verify"
    },
    "strict": {
      "type": "boolean",
      "description": "If true, every sentence in the answer must be supported (default: false)"
    },
    "context": {
      "type": "string",
      "description": "The source/reference text that should ground the answer"
    }
  }
}

list_llm_models

List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers.

read-only

Argument	Type	Description
provider	string	Filter by provider name (case-insensitive). E.g. "Groq", "HuggingFace", "OpenAI", "Anthropic", "Google", "DeepSeek", "xAI", "Ollama". Omit for full catalog.

Input schema (JSON)

{
  "type": "object",
  "properties": {
    "provider": {
      "type": "string",
      "description": "Filter by provider name (case-insensitive). E.g. \"Groq\", \"HuggingFace\", \"OpenAI\", \"Anthropic\", \"Google\", \"DeepSeek\", \"xAI\", \"Ollama\". Omit for full catalog."
    }
  }
}

llm_format_check

Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.

read-only

Argument	Type	Description
output*	string	The LLM output to validate
regex_pattern	string	Custom regex pattern (only when expected_format is "regex")
expected_format*	string	Expected format

Input schema (JSON)

{
  "type": "object",
  "required": [
    "output",
    "expected_format"
  ],
  "properties": {
    "output": {
      "type": "string",
      "description": "The LLM output to validate"
    },
    "regex_pattern": {
      "type": "string",
      "description": "Custom regex pattern (only when expected_format is \"regex\")"
    },
    "expected_format": {
      "enum": [
        "json",
        "markdown_heading",
        "code_block",
        "bullet_list",
        "numbered_list",
        "table",
        "yaml",
        "xml",
        "regex"
      ],
      "type": "string",
      "description": "Expected format"
    }
  }
}

llm_generate

Generate text using open-source LLM models hosted on Groq (ultra-fast) or HuggingFace Inference (serverless). No API key required — the server provides its own keys. Supported models: Qwen3 32B, Gemma 4 27B, Gemma 3 27B, Llama 3.3 70B, Llama 4 Scout, DeepSeek R1, Mistral Small 24B, and more. Use list_llm_models to see the full catalog. Rate-limited to prevent abuse.

read-only

Argument	Type	Description
model	string	Model ID (default: "openai/gpt-oss-20b"). Server-keyed whitelist only — Groq: openai/gpt-oss-20b, openai/gpt-oss-120b; HuggingFace: Qwen/Qwen3-32B, meta-llama/Llama-3.3-70B-Instruct, deepseek-ai/DeepSeek-R1, google/gemma-3-27b-it, and more. Other ids from list_llm_models are BYOK-only and will be rejected.
prompt*	string	The user prompt / instruction to send to the model
system	string	Optional system prompt to set context or persona
max_tokens	number	Maximum tokens to generate (default: 2048, max: 4096)
temperature	number	Sampling temperature 0.0–1.5 (default: 0.7)

Input schema (JSON)

{
  "type": "object",
  "required": [
    "prompt"
  ],
  "properties": {
    "model": {
      "type": "string",
      "description": "Model ID (default: \"openai/gpt-oss-20b\"). Server-keyed whitelist only — Groq: openai/gpt-oss-20b, openai/gpt-oss-120b; HuggingFace: Qwen/Qwen3-32B, meta-llama/Llama-3.3-70B-Instruct, deepseek-ai/DeepSeek-R1, google/gemma-3-27b-it, and more. Other ids from list_llm_models are BYOK-only and will be rejected."
    },
    "prompt": {
      "type": "string",
      "description": "The user prompt / instruction to send to the model"
    },
    "system": {
      "type": "string",
      "description": "Optional system prompt to set context or persona"
    },
    "max_tokens": {
      "type": "number",
      "description": "Maximum tokens to generate (default: 2048, max: 4096)"
    },
    "temperature": {
      "type": "number",
      "description": "Sampling temperature 0.0–1.5 (default: 0.7)"
    }
  }
}

llm_json_schema_check

Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.

read-only

Argument	Type	Description
output*	string	The LLM JSON output (raw string, will be parsed)
schema*	object	JSON Schema (draft-07 subset) to validate against

Input schema (JSON)

{
  "type": "object",
  "required": [
    "output",
    "schema"
  ],
  "properties": {
    "output": {
      "type": "string",
      "description": "The LLM JSON output (raw string, will be parsed)"
    },
    "schema": {
      "type": "object",
      "description": "JSON Schema (draft-07 subset) to validate against",
      "additionalProperties": true
    }
  }
}

llm_output_validator

Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.

read-only

Argument	Type	Description
output*	string	The LLM output text to validate
max_length	number	Maximum character length for the output
min_length	number	Minimum character length for the output
check_safety	boolean	Check for PII patterns (emails, phones, SSN), profanity signals, and prompt leakage
must_include	string	Comma-separated strings that MUST appear in the output
expected_format	string	Expected output format
must_not_include	string	Comma-separated strings that must NOT appear (e.g. "TODO, FIXME, undefined, NaN")
check_json_schema	string	If expected_format is JSON, provide required keys as comma-separated list to validate the structure

metamorphic_check

Reference-free stability primitive: instead of comparing an answer to a ground truth, it checks that an assistant's answer stays INVARIANT when the QUESTION is transformed (typo, casing, paraphrase, reordering, translation). Catches the failure class no reference answer can expose — an assistant that handles one phrasing well and a trivial variant of it badly. You bring the outputs (no model is called), so it is deterministic and free in tfidf mode. Relations: case (θ .95), typo (.90), paraphrase (.80), reorder (.80), translation (.75, embeddings only), specialization (.60, ADVISORY — directional, never gated). Returns PASS / FAIL / INVALID, where INVALID means the BASE answer was a refusal or too short so invariance was never measurable — an assistant that refuses every variant would otherwise score a perfect 1.0. Use run_semantic_tests alongside it: invariance without a correctness floor is a green light for a broken assistant.

read-only

Argument	Type	Description
base*	object	The reference run: the original question and the answer your system produced for it.
mode	string	tfidf (default): free, lexical, deterministic — but a genuine paraphrase rarely reaches 0.80, so gate on case/typo and treat paraphrase as a trend. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key. translation requires this mode.
api_key	string	OpenAI API key — required only when mode is embeddings.
variants*	array	Answers produced for transformed versions of the same question, each tagged with the relation that was applied.

model_info

Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.

read-only

Argument	Type	Description
model*	string	Model name (e.g. "gpt-4o", "claude-3.5-sonnet", "gemini-3.1-pro-preview")

Input schema (JSON)

{
  "type": "object",
  "required": [
    "model"
  ],
  "properties": {
    "model": {
      "type": "string",
      "description": "Model name (e.g. \"gpt-4o\", \"claude-3.5-sonnet\", \"gemini-3.1-pro-preview\")"
    }
  }
}

needle_haystack_generate

Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.

read-only

Argument	Type	Description
needle*	string	The fact to hide (e.g. "The secret code is ALPHA-42")
tokens	integer	Target haystack size in tokens (default: 5000, max: 100000)
position	string	Where to insert the needle: "start", "middle", "end", "random" (default: "middle")
question*	string	The question to ask the LLM (e.g. "What is the secret code?")

Input schema (JSON)

{
  "type": "object",
  "required": [
    "needle",
    "question"
  ],
  "properties": {
    "needle": {
      "type": "string",
      "description": "The fact to hide (e.g. \"The secret code is ALPHA-42\")"
    },
    "tokens": {
      "type": "integer",
      "default": 5000,
      "description": "Target haystack size in tokens (default: 5000, max: 100000)"
    },
    "position": {
      "enum": [
        "start",
        "middle",
        "end",
        "random"
      ],
      "type": "string",
      "default": "middle",
      "description": "Where to insert the needle: \"start\", \"middle\", \"end\", \"random\" (default: \"middle\")"
    },
    "question": {
      "type": "string",
      "description": "The question to ask the LLM (e.g. \"What is the secret code?\")"
    }
  }
}

prompt_test_suite

Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.

read-only

Argument	Type	Description
max_tokens	number	Max token budget for the test
temperature	number	Temperature to use
user_prompt*	string	The user prompt to send
check_safety	boolean	Include safety/PII checks in the rubric
must_include	string	Required content (comma-separated)
system_prompt*	string	The system prompt under test
expected_format	string	Expected output format
must_not_include	string	Forbidden content (comma-separated)
expected_behavior	string

rag_relevance_rank

Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.

read-only

Argument	Type	Description
query*	string	The user query
top_k	number	Return top K results (default: all)
chunks*	array	Array of text chunks to rank

Input schema (JSON)

{
  "type": "object",
  "required": [
    "query",
    "chunks"
  ],
  "properties": {
    "query": {
      "type": "string",
      "description": "The user query"
    },
    "top_k": {
      "type": "number",
      "description": "Return top K results (default: all)"
    },
    "chunks": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Array of text chunks to rank"
    }
  }
}

rate_tool

Give honest usage feedback on an IA-QA MCP tool. Provide a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Ratings are aggregated on a public dashboard at /devtools/mcp-ratings. Skip rating routine successes — we want signal, not praise. Example: rate_tool({ tool_name: "format_json", score: 2, comment: "Tried to pretty-print a JSON5 file, it rejected trailing commas — not usable for my case." })

Argument	Type	Description
score*	number	Rating from 1 (poor) to 5 (excellent)
comment	string	Strongly encouraged — explain what you were trying to do and whether the tool got you there. Be specific about what was missing, wrong, or a poor fit. This is the most valuable part of the rating. Up to 2000 chars are stored; go over and the response says so (truncated: true) — send the remainder as a second call rather than assuming it landed.
tool_name*	string	Name of the MCP tool to rate (e.g. "format_json", "shield_analyze")

Input schema (JSON)

{
  "type": "object",
  "required": [
    "tool_name",
    "score"
  ],
  "properties": {
    "score": {
      "type": "number",
      "maximum": 5,
      "minimum": 1,
      "description": "Rating from 1 (poor) to 5 (excellent)"
    },
    "comment": {
      "type": "string",
      "description": "Strongly encouraged — explain what you were trying to do and whether the tool got you there. Be specific about what was missing, wrong, or a poor fit. This is the most valuable part of the rating. Up to 2000 chars are stored; go over and the response says so (truncated: true) — send the remainder as a second call rather than assuming it landed."
    },
    "tool_name": {
      "type": "string",
      "description": "Name of the MCP tool to rate (e.g. \"format_json\", \"shield_analyze\")"
    }
  }
}

rerank_evaluate

Evaluate RAG retrieval quality: rank passages against a query and compute Precision@k / Recall@k plus a PASS/FAIL CI verdict from ground-truth relevance labels. Three modes, all keyless except the last. (1) BYO scores — give each passage the `score` your own reranker produced (Cohere, Jina, a self-hosted NIM, a cross-encoder): deterministic, offline, and it evaluates YOUR reranker rather than someone else's. This is the mode to gate CI on. (2) Default, no scores and no key — ranks with local BM25, a lexical keyword baseline: it answers "does a keyword floor already surface my relevant passages?", never "is my neural reranker good". (3) Live NVIDIA reranker — supply `api_key` for an NVIDIA account that still has reranking entitlement; NVIDIA retired its hosted reranking endpoints on 2026-05-18, so this one is for accounts that were grandfathered in.

read-only

Argument	Type	Description
query*	string	The search query or question to rank against
top_k	integer	k for Precision@k evaluation (default 3)
api_key	string	Your NVIDIA API key (BYOK), used only when no passage carries a score. Transits RAM for the single call, never stored.
passages*	array	Array of passage objects to rank (min 2, max 20)
threshold	number	Minimum Precision@k to PASS (0-1, default 0.5)

Input schema (JSON)

response_quality_score

Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a correct one-word answer shares no words with its question). For meaning against a reference answer, use run_semantic_tests. Also returns unscored signals: question-term overlap, average sentence length, markdown.

read-only

Argument	Type	Description
question*	string	The original question/prompt
response*	string	The LLM response to score
max_length	number	Maximum character length. Scored as a proportional penalty beyond it.
expected_keywords	array	Keywords a correct answer must contain (case-insensitive). Scored as coverage.

Input schema (JSON)

{
  "type": "object",
  "required": [
    "question",
    "response"
  ],
  "properties": {
    "question": {
      "type": "string",
      "description": "The original question/prompt"
    },
    "response": {
      "type": "string",
      "description": "The LLM response to score"
    },
    "max_length": {
      "type": "number",
      "description": "Maximum character length. Scored as a proportional penalty beyond it."
    },
    "expected_keywords": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Keywords a correct answer must contain (case-insensitive). Scored as coverage."
    }
  }
}

run_eval_contract

Parse a .ia-eval.yaml LLM test suite, call the specified LLM model for each scenario, run all configured scorers, and return a structured JSON report with per-scenario Pass/Fail verdicts and a Markdown summary. Use list_local_tests to discover available test files.

read-only

Argument	Type	Description
api_keys	object	API keys to use for LLM generation (all optional — falls back to server env vars)
overrides	object	Override contract defaults
contract_path	string	Absolute or relative path to a .ia-eval.yaml file (required unless inline_contract is provided)
validate_only	boolean	Check the contract against the IA-EVAL schema and stop: no model call, no key, deterministic. Returns valid + issues instead of throwing. Gate a PR that edits a contract on valid.
inline_contract	object	Raw contract object (alternative to contract_path). Must contain top-level "metadata" ({name, version, model?, provider?}), "expectations" ({min_score?}), and "scenarios" ([{id, input, ground_truth?}]) — scenarios alone are rejected. Use generate_eval_yaml to scaffold one.

Input schema (JSON)

{
  "type": "object",
  "properties": {
    "api_keys": {
      "type": "object",
      "properties": {
        "hf": {
          "type": "string"
        },
        "groq": {
          "type": "string"
        },
        "google": {
          "type": "string"
        },
        "openai": {
          "type": "string"
        },
        "anthropic": {
          "type": "string"
        }
      },
      "description": "API keys to use for LLM generation (all optional — falls back to server env vars)"
    },
    "overrides": {
      "type": "object",
      "properties": {
        "model": {
          "type": "string"
        },
        "provider": {
          "type": "string"
        },
        "temperature": {
          "type": "number"
        },
        "system_prompt": {
          "type": "string"
        }
      },
      "description": "Override contract defaults"
    },
    "contract_path": {
      "type": "string",
      "description": "Absolute or relative path to a .ia-eval.yaml file (required unless inline_contract is provided)"
    },
    "validate_only": {
      "type": "boolean",
      "description": "Check the contract against the IA-EVAL schema and stop: no model call, no key, deterministic. Returns valid + issues instead of throwing. Gate a PR that edits a contract on valid."
    },
    "inline_contract": {
      "type": "object",
      "description": "Raw contract object (alternative to contract_path). Must contain top-level \"metadata\" ({name, version, model?, provider?}), \"expectations\" ({min_score?}), and \"scenarios\" ([{id, input, ground_truth?}]) — scenarios alone are rejected. Use generate_eval_yaml to scaffold one.",
      "additionalProperties": true
    }
  }
}

run_semantic_tests

Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.

read-only

Argument	Type	Description
mode	string	tfidf (default): fast, free, lexical. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key.
cases*	array	Array of (actual, expected) pairs to evaluate.
api_key	string	OpenAI API key — required only when mode is embeddings.
thresholds	object	Pass/fail thresholds (defaults: cosine 0.75, rouge_l 0.5).
require_all	boolean	If true (default), all cases must pass for overall PASS. If false, at least one case passing returns PASS.

Input schema (JSON)

{
  "type": "object",
  "required": [
    "cases"
  ],
  "properties": {
    "mode": {
      "enum": [
        "tfidf",
        "embeddings"
      ],
      "type": "string",
      "description": "tfidf (default): fast, free, lexical. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key."
    },
    "cases": {
      "type": "array",
      "items": {
        "type": "object",
        "required": [
          "actual",
          "expected"
        ],
        "properties": {
          "id": {
            "type": "string",
            "description": "Optional identifier for this case."
          },
          "actual": {
            "type": "string",
            "description": "The text produced by your LLM/system."
          },
          "expected": {
            "type": "string",
            "description": "The reference/ground-truth text."
          }
        }
      },
      "maxItems": 50,
      "description": "Array of (actual, expected) pairs to evaluate."
    },
    "api_key": {
      "type": "string",
      "description": "OpenAI API key — required only when mode is embeddings."
    },
    "thresholds": {
      "type": "object",
      "properties": {
        "cosine": {
          "type": "number",
          "maximum": 1,
          "minimum": 0,
          "description": "Minimum cosine similarity to pass (default: 0.75)."
        },
        "rouge_l": {
          "type": "number",
          "maximum": 1,
          "minimum": 0,
          "description": "Minimum ROUGE-L F1 to pass (default: 0.5)."
        }
      },
      "description": "Pass/fail thresholds (defaults: cosine 0.75, rouge_l 0.5)."
    },
    "require_all": {
      "type": "boolean",
      "description": "If true (default), all cases must pass for overall PASS. If false, at least one case passing returns PASS."
    }
  }
}

run_vlm_test_suite

Run a test suite against a Vision-Language Model (VLM) — send an image (URL or base64) + N test cases (each with a question + assertion) to GPT-4o, Claude 3.5, or Gemini. Returns per-case PASS/FAIL verdicts, a pass rate, an overall PASS/WARNING/FAIL verdict (customizable threshold), and latency stats. Assertion types: contains, not_contains, json_format, min_length, max_length, semantic_contains (TF-IDF cosine similarity ≥ 0.4). BYOK: requires your own API key for the target provider.

read-only

Argument	Type	Description
model*	string	VLM model to use.
api_key*	string	API key for the model provider (OpenAI sk-, Anthropic sk-ant-, or Google AIzaSy...).
image_url	string	Public URL of the image to evaluate (required unless image_base64 is provided).
threshold	number	Pass rate threshold for overall verdict (default: 80, 0–100).
test_cases*	array	Array of test cases to run.
image_base64	string	Base64-encoded image data (required unless image_url is provided).
system_prompt	string

similarity_score

Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.

read-only

Argument	Type	Description
batch	array	Batch mode: array of {reference, hypothesis} pairs.
metrics	array	Metrics to compute (default: all). Options: "cosine_bow", "cosine_tfidf", "jaccard", "rouge1", "rouge2", "rougeL", "bleu"
reference	string	Reference / expected text (ground truth)
threshold	number	Optional pass/fail threshold (0-1). Applies to ROUGE-L F1 score.
hypothesis	string	Hypothesis / actual text (LLM output)

Input schema (JSON)

{
  "type": "object",
  "properties": {
    "batch": {
      "type": "array",
      "items": {
        "type": "object",
        "required": [
          "reference",
          "hypothesis"
        ],
        "properties": {
          "reference": {
            "type": "string"
          },
          "hypothesis": {
            "type": "string"
          }
        }
      },
      "description": "Batch mode: array of {reference, hypothesis} pairs."
    },
    "metrics": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Metrics to compute (default: all). Options: \"cosine_bow\", \"cosine_tfidf\", \"jaccard\", \"rouge1\", \"rouge2\", \"rougeL\", \"bleu\""
    },
    "reference": {
      "type": "string",
      "description": "Reference / expected text (ground truth)"
    },
    "threshold": {
      "type": "number",
      "description": "Optional pass/fail threshold (0-1). Applies to ROUGE-L F1 score."
    },
    "hypothesis": {
      "type": "string",
      "description": "Hypothesis / actual text (LLM output)"
    }
  }
}

test_skill

Test a SKILL.md in the open Agent Skills format (https://agentskills.io/specification — read by Claude Code, Codex, GitHub Copilot, Cursor, Gemini CLI, OpenCode and others) in two stages. (1) lint: the spec's frontmatter rules checked without a model (name 1-64 chars, lowercase/digits/single hyphens, matches dir_name when given; description 1-1024 chars; compatibility ≤500; metadata string map) — any error returns BLOCK and calls no model; gate on lint.valid, it is deterministic. (2) triggering, measured the way an agent decides: the model sees ONLY the name + description of this skill and of the other installed skills (other_skills, or six built-in neighbours) and answers which one it would load, at temperature 0. trigger_accuracy = share of requests routed correctly (should load / should not); verdict PASS ≥80, FIX ≥50, else BLOCK. Pass your own scenarios [{prompt, should_trigger}] for a reproducible gate; without them requests are generated (half should-load, half near misses) and the result is a signal, flagged by scenarios_source. step_adherence is keyword overlap on the skill body's run and never enters the verdict. A request the model never answered readably (rate limit, empty output) is not counted as a miss: the run is INCONCLUSIVE instead. Uses the server's default Groq model (server key, rate-limited). Pass api_key + model to use your own provider; the SKILL.md and scenarios are sent to ia-qa.com and to that provider.

read-only

Argument	Type	Description
model	string	LLM model ID to use for both scenario generation and testing (e.g. gpt-4o-mini, claude-3-5-haiku-20241022). Defaults to openai/gpt-oss-20b (Groq, server key).
api_key	string	API key for the chosen model provider. Not required when using the default Groq model.
dir_name	string

token_budget_calculator

Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds model context window. Supports 25+ models.

read-only

Argument	Type	Description
model*	string	Model name (e.g. gpt-4o, claude-3.5-sonnet, gemini-3.5-flash)
context	string	Actual context text (will estimate tokens)
user_input	string	Actual user input text (will estimate tokens)
system_prompt	string	Actual system prompt text (will estimate tokens)
context_tokens	number	Token count for RAG context / documents
user_input_tokens	number	Token count for user message
system_prompt_tokens	number	Token count for system prompt
expected_output_tokens	number	Expected max output tokens

Input schema (JSON)

{
  "type": "object",
  "properties": {
    "cron": {
      "type": "string",
      "description": "Cron expression when triggers include 'schedule' (default: '0 6 * * 1' — Mondays 06:00 UTC)."
    },
    "gate": {
      "enum": [
        "eval_contract",
        "cli_checks",
        "both",
        "selector_drift",
        "all"
      ],
      "type": "string",
      "description": "Which gate to emit. eval_contract = LLM eval via the action (default). cli_checks = deterministic CLI assertions. selector_drift = an E2E selector-drift gate via @ia-qa/self-healing (boots the app, captures, diffs against the committed baseline, branches on exit code 0/1/2). both = CLI checks + eval. all = CLI checks, then drift, then eval."
    },
    "provider": {
      "enum": [
        "groq",
        "openai",
        "anthropic",
        "google"
      ],
      "type": "string",
      "description": "LLM provider the contract runs against — decides which repository secret the workflow wires (default: groq)."
    },
    "triggers": {
      "type": "array",
      "items": {
        "enum": [
          "push",
          "pull_request",
          "workflow_dispatch",
          "schedule"
        ],
        "type": "string"
      },
      "description": "Workflow triggers (default: push + pull_request)."
    },
    "cli_tools": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "IA-QA tool names to run as deterministic gates, e.g. [\"secret_scan\",\"prompt_injection_scan\"]. Tools with no known CI recipe get a --stdin step flagged in notes."
    },
    "min_score": {
      "type": "number",
      "description": "Override the contract min_score (0-100). Omit to use the value in the contract."
    },
    "app_base_url": {
      "type": "string",
      "description": "URL the drift gate waits for before capturing (default: http://127.0.0.1:3000). Must match config.baseUrl in .ia-qa/config.json."
    },
    "fail_on_fail": {
      "type": "boolean",
      "description": "Fail the build on a FAIL/PARTIAL verdict (default: true). Set false to report without gating."
    },
    "node_version": {
      "type": "string",
      "description": "Node version for the CLI steps (default: \"20\")."
    },
    "contract_path": {
      "type": "string",
      "description": "Path to the .ia-eval.yaml contract, relative to the repo root (default: evals/smoke.ia-eval.yaml). Only used when the gate includes eval_contract."
    },
    "workflow_name": {
      "type": "string",
      "description": "Workflow display name (default: \"IA-QA Quality Gate\")."
    },
    "app_start_command": {
      "type": "string",
      "description": "Command that boots the app for the selector_drift gate, e.g. \"npm run start:ci\". Cannot be guessed — omitted, the step carries an explicit TODO and the note says so, because a plausible default would silently map nothing."
    }
  }
}

Input schema (JSON)

{
  "type": "object",
  "required": [
    "base",
    "variants"
  ],
  "properties": {
    "base": {
      "type": "object",
      "properties": {
        "output": {
          "type": "string",
          "description": "Required — the answer your system produced for the original question."
        },
        "question": {
          "type": "string",
          "description": "Optional — the original question, echoed back in the output for readability."
        }
      },
      "description": "The reference run: the original question and the answer your system produced for it."
    },
    "mode": {
      "enum": [
        "tfidf",
        "embeddings"
      ],
      "type": "string",
      "description": "tfidf (default): free, lexical, deterministic — but a genuine paraphrase rarely reaches 0.80, so gate on case/typo and treat paraphrase as a trend. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key. translation requires this mode."
    },
    "api_key": {
      "type": "string",
      "description": "OpenAI API key — required only when mode is embeddings."
    },
    "variants": {
      "type": "array",
      "items": {
        "type": "object",
        "required": [
          "relation",
          "output"
        ],
        "properties": {
          "id": {
            "type": "string",
            "description": "Optional identifier (default: variant_<n>)."
          },
          "output": {
            "type": "string",
            "description": "Required — the answer your system produced for the transformed question."
          },
          "question": {
            "type": "string",
            "description": "Optional — the transformed question, echoed back for readability."
          },
          "relation": {
            "enum": [
              "case",
              "typo",
              "paraphrase",
              "reorder",
              "translation",
              "specialization"
            ],
            "type": "string",
            "description": "Transformation applied to the QUESTION. Decides the metric and the default threshold."
          }
        }
      },
      "maxItems": 20,
      "description": "Answers produced for transformed versions of the same question, each tagged with the relation that was applied."
    },
    "thresholds": {
      "type": "object",
      "properties": {
        "case": {
          "type": "number",
          "maximum": 1,
          "minimum": 0
        },
        "typo": {
          "type": "number",
          "maximum": 1,
          "minimum": 0
        },
        "reorder": {
          "type": "number",
          "maximum": 1,
          "minimum": 0
        },
        "paraphrase": {
          "type": "number",
          "maximum": 1,
          "minimum": 0
        },
        "translation": {
          "type": "number",
          "maximum": 1,
          "minimum": 0
        },
        "specialization": {
          "type": "number",
          "maximum": 1,
          "minimum": 0
        }
      },
      "description": "Per-relation threshold overrides. Calibrate on your own corpus before gating — the defaults are starting points, not measurements."
    },
    "require_all": {
      "type": "boolean",
      "description": "If true (default), every gated variant must pass. KEEP THE DEFAULT for any run you gate on. Setting it false is not a tolerance dial but an off switch: relations have asymmetric pass rates (a typo variant usually scores ~1.0 because the answer really is identical), so one trivial row is enough to hold the whole run at PASS while a paraphrase fails. When that happens the result carries an explicit warning naming the failed rows."
    },
    "baseline_guard": {
      "type": "object",
      "properties": {
        "min_length": {
          "type": "number",
          "description": "Minimum base answer length in chars (default 40)."
        },
        "must_not_match": {
          "type": "array",
          "items": {
            "type": "string"
          },
          "description": "Refusal patterns, matched case-insensitively in the FIRST 200 CHARS of the base answer (refusals lead; matching anywhere would flag a long correct answer that merely mentions one). Replaces the FR+EN default list, never merges — pass [] to disable."
        }
      },
      "description": "Correctness floor applied to the BASE answer before anything is scored. Failing it returns INVALID, not FAIL."
    }
  }
}

{
  "type": "object",
  "required": [
    "query",
    "passages"
  ],
  "properties": {
    "query": {
      "type": "string",
      "description": "The search query or question to rank against"
    },
    "top_k": {
      "type": "integer",
      "maximum": 10,
      "minimum": 1,
      "description": "k for Precision@k evaluation (default 3)"
    },
    "api_key": {
      "type": "string",
      "description": "Your NVIDIA API key (BYOK), used only when no passage carries a score. Transits RAM for the single call, never stored."
    },
    "passages": {
      "type": "array",
      "items": {
        "type": "object",
        "required": [
          "text"
        ],
        "properties": {
          "id": {
            "type": "string"
          },
          "text": {
            "type": "string"
          },
          "score": {
            "type": "number",
            "description": "Relevance score from YOUR reranker. Present on every passage → ranking is done from these (offline, no key). Higher = more relevant. Score all passages or none."
          },
          "relevant": {
            "type": "boolean",
            "description": "Ground truth: is this passage relevant?"
          }
        }
      },
      "description": "Array of passage objects to rank (min 2, max 20)"
    },
    "threshold": {
      "type": "number",
      "maximum": 1,
      "minimum": 0,
      "description": "Minimum Precision@k to PASS (0-1, default 0.5)"
    }
  }
}

Input schema (JSON)

{
  "type": "object",
  "required": [
    "test_cases",
    "model",
    "api_key"
  ],
  "properties": {
    "model": {
      "enum": [
        "gpt-4o",
        "gpt-4o-mini",
        "claude-3-5-sonnet-20241022",
        "claude-3-5-haiku-20241022",
        "gemini-3.5-flash",
        "gemini-2.5-flash"
      ],
      "type": "string",
      "description": "VLM model to use."
    },
    "api_key": {
      "type": "string",
      "description": "API key for the model provider (OpenAI sk-, Anthropic sk-ant-, or Google AIzaSy...)."
    },
    "image_url": {
      "type": "string",
      "description": "Public URL of the image to evaluate (required unless image_base64 is provided)."
    },
    "threshold": {
      "type": "number",
      "description": "Pass rate threshold for overall verdict (default: 80, 0–100)."
    },
    "test_cases": {
      "type": "array",
      "items": {
        "type": "object",
        "required": [
          "question"
        ],
        "properties": {
          "id": {
            "type": "string",
            "description": "Optional identifier for this case."
          },
          "question": {
            "type": "string",
            "description": "Question to ask the VLM about the image."
          },
          "assertion_type": {
            "enum": [
              "contains",
              "not_contains",
              "json_format",
              "min_length",
              "max_length",
              "semantic_contains"
            ],
            "type": "string",
            "description": "Assertion to run on the VLM response. semantic_contains uses TF-IDF cosine similarity ≥ 0.4."
          },
          "assertion_value": {
            "type": "string",
            "description": "Expected value for the assertion (not needed for json_format)."
          }
        }
      },
      "maxItems": 10,
      "description": "Array of test cases to run."
    },
    "image_base64": {
      "type": "string",
      "description": "Base64-encoded image data (required unless image_url is provided)."
    },
    "system_prompt": {
      "type": "string",
      "description": "Optional system prompt sent to the VLM."
    },
    "image_mime_type": {
      "type": "string",
      "description": "MIME type of the image if using image_base64 (default: image/jpeg)."
    }
  }
}

Input schema (JSON)

{
  "type": "object",
  "required": [
    "skill_md"
  ],
  "properties": {
    "model": {
      "type": "string",
      "description": "LLM model ID to use for both scenario generation and testing (e.g. gpt-4o-mini, claude-3-5-haiku-20241022). Defaults to openai/gpt-oss-20b (Groq, server key)."
    },
    "api_key": {
      "type": "string",
      "description": "API key for the chosen model provider. Not required when using the default Groq model."
    },
    "dir_name": {
      "type": "string",
      "description": "Name of the folder holding the SKILL.md. The spec requires name to match it; checked only when given."
    },
    "skill_md": {
      "type": "string",
      "description": "Full content of the SKILL.md file to test, YAML frontmatter included."
    },
    "lint_only": {
      "type": "boolean",
      "description": "Run the format check only: no model call, no key, deterministic. The CI gate for a SKILL.md — gate on lint.valid."
    },
    "scenarios": {
      "type": "array",
      "items": {
        "type": "object",
        "required": [
          "prompt",
          "should_trigger"
        ],
        "properties": {
          "prompt": {
            "type": "string"
          },
          "should_trigger": {
            "type": "boolean"
          }
        }
      },
      "description": "Fixed user requests for a reproducible gate: [{\"prompt\": \"...\", \"should_trigger\": true}]. Include near misses with should_trigger false. Max 20. When omitted, scenario_count requests are generated."
    },
    "other_skills": {
      "type": "array",
      "items": {
        "type": "object",
        "required": [
          "name",
          "description"
        ],
        "properties": {
          "name": {
            "type": "string"
          },
          "description": {
            "type": "string"
          }
        }
      },
      "description": "The other skills installed next to this one, [{\"name\", \"description\"}], so triggering is tested against the real neighbours. Default: six built-in generic skills."
    },
    "scenario_count": {
      "enum": [
        4,
        6,
        8,
        10
      ],
      "type": "number",
      "description": "Number of requests to generate when scenarios is omitted: half should-load, half near misses. Default: 6."
    }
  }
}

{
  "type": "object",
  "required": [
    "model"
  ],
  "properties": {
    "model": {
      "type": "string",
      "description": "Model name (e.g. gpt-4o, claude-3.5-sonnet, gemini-3.5-flash)"
    },
    "context": {
      "type": "string",
      "description": "Actual context text (will estimate tokens)"
    },
    "user_input": {
      "type": "string",
      "description": "Actual user input text (will estimate tokens)"
    },
    "system_prompt": {
      "type": "string",
      "description": "Actual system prompt text (will estimate tokens)"
    },
    "context_tokens": {
      "type": "number",
      "description": "Token count for RAG context / documents"
    },
    "user_input_tokens": {
      "type": "number",
      "description": "Token count for user message"
    },
    "system_prompt_tokens": {
      "type": "number",
      "description": "Token count for system prompt"
    },
    "expected_output_tokens": {
      "type": "number",
      "description": "Expected max output tokens"
    }
  }
}