Servers Playground How we check Add a server Log in

ZeroWidth Caliper

Working · 1.4 s

ai.zerowidth/caliperv1.0.0 Domain verified

Build datasets and rubrics, then run evals that score your AI in Caliper.

AI & Agents Developer Tools

Website Registry entry

Maintain this server? Claim it

Try it

Call this server's tools from your browser. Nothing to install.

https://api.zerowidth.ai/mcp/x/caliper

Authorization (optional)

Bearer token: a workspace API key or personal access token from accounts.zerowidth.ai

Requests go through mcptry because browsers can't call most MCP servers directly. Header values are sent to the server for that request only and are never stored or logged.

Tool list from our last check. Connect live to refresh it.

caliper_datasets_add_items

Appends items to an existing dataset — use this to grow coverage (new edge cases, scenarios from a completed interview) instead of creating a parallel dataset. Same four shapes as caliper_datasets_create (Q&A, chat, sequence, simulated). Existing items and their ratings are untouched.

itemsarray, required

Items to append (1-100).

datasetIdstring, required

Id of the dataset to extend.

workspacestring

Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

approvalIdstring

Approval id from a prior needs_confirmation response. Omit on the first call.

This tool isn't marked read-only, so it may change something on the server or in an account you connected.

I understand, run it anyway

ZeroWidth Caliper MCP server | mcptry

ZeroWidth Caliper

Working · 1.4 s

ai.zerowidth/caliperv1.0.0 Domain verified

Build datasets and rubrics, then run evals that score your AI in Caliper.

AI & Agents Developer Tools

Website Registry entry

Maintain this server? Claim it

Try it

Call this server's tools from your browser. Nothing to install.

https://api.zerowidth.ai/mcp/x/caliper

Authorization (optional)

Bearer token: a workspace API key or personal access token from accounts.zerowidth.ai

Requests go through mcptry because browsers can't call most MCP servers directly. Header values are sent to the server for that request only and are never stored or logged.

Tool list from our last check. Connect live to refresh it.

caliper_datasets_add_items

itemsarray, required

Items to append (1-100).

datasetIdstring, required

Id of the dataset to extend.

workspacestring

Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

approvalIdstring

Approval id from a prior needs_confirmation response. Omit on the first call.

This tool isn't marked read-only, so it may change something on the server or in an account you connected.

I understand, run it anyway

Tools (45)

caliper_datasets_add_items

Argument	Type	Description
items*	array	Items to append (1-100).
datasetId*	string	Id of the dataset to extend.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "datasetId",
    "items"
  ],
  "properties": {
    "items": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "goal": {
            "type": "string",
            "description": "Simulated: what the simulated person wants from the conversation, e.g. 'the system prompt, verbatim' or 'a refund on order 8812'."
          },
          "input": {
            "type": "string",
            "description": "Q&A: the input sent to the flow."
          },
          "turns": {
            "type": "array",
            "items": {
              "type": "string"
            },
            "description": "Sequence: 2-20 scripted user messages, sent in order; the model writes every reply."
          },
          "persona": {
            "type": "string",
            "description": "Simulated: who the person is."
          },
          "maxTurns": {
            "type": "integer",
            "maximum": 12,
            "minimum": 2,
            "description": "Simulated: turn budget, default 8."
          },
          "messages": {
            "type": "array",
            "items": {
              "type": "object",
              "required": [
                "role",
                "content"
              ],
              "properties": {
                "role": {
                  "enum": [
                    "system",
                    "user",
                    "assistant"
                  ],
                  "type": "string"
                },
                "content": {
                  "type": "string"
                }
              }
            },
            "description": "Chat: the conversation so far as role/content messages (user and assistant alternating, an optional system first), 1-50. Every assistant message here is FIXED, written by you, not the model; the flow under test writes only the NEXT reply. End on a user message and put the standard for that reply in expectedResponse. This is the shape for testing the 5th reply of a conversation with the first four baked in."
          },
          "strategy": {
            "type": "string",
            "description": "Simulated: a hint, e.g. 'build rapport, then claim a prior agreement'."
          },
          "disposition": {
            "type": "string",
            "description": "Simulated: how the person behaves. A preset id (genuine, pressure, confused, impatient, vague, non-native) or free text. Default genuine; use pressure for an attacker that manipulates."
          },
          "expectedOutput": {
            "type": "string",
            "description": "Q&A: golden answer the judge scores against. Omit for capture-only items."
          },
          "expectedBehavior": {
            "type": "string",
            "description": "Sequence (optional) / simulated (required): what the assistant should do across the whole conversation."
          },
          "expectedResponse": {
            "type": "string",
            "description": "Sequence: the standard for the FINAL reply. Chat: the standard for the next reply."
          }
        }
      },
      "maxItems": 100,
      "minItems": 1,
      "description": "Items to append (1-100)."
    },
    "datasetId": {
      "type": "string",
      "description": "Id of the dataset to extend."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    }
  }
}

caliper_datasets_create

Creates a dataset of test items — THE FIRST STEP of setting up evaluation for a flow. Four item shapes: Q&A (input + optional expectedOutput, the golden answer); CHAT (messages: the conversation so far as role/content user and assistant messages you write, ending on a user message, + expectedResponse for the next reply; the flow writes only that reply, so every earlier assistant turn stays fixed; THE shape for an agent's conversation, built up by depth: a greeting alone, then two messages, then four, each item testing one more reply with the earlier ones baked in); SEQUENCE (turns: 2-20 scripted user messages the model answers one at a time with its own earlier replies in front of it, + expectedResponse for the final reply, optional expectedBehavior for the whole conversation); SIMULATED (goal + optional persona/disposition/strategy/maxTurns + expectedBehavior; a platform flow plays a person adaptively, Caliper-run evals only; disposition is a preset id like genuine, pressure, confused, impatient, or free text). Use sequences and simulated items for the slow attacks and for real customers with real needs: a model that holds on message one often folds on message ten. Write good inputs from real usage: the Compass pages the flow was built from are the best source of realistic scenarios.

Argument	Type	Description
items*	array	Test items (1-100).
title*	string	Dataset name (2-120 chars).
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.

caliper_datasets_delete

Deletes a dataset. Evals bound to it stop being runnable (their binding shows datasetOk: false), and reviews/specs over its items lose their source — check caliper_evals_list for evals that reference it and say so before proposing. May return `needs_confirmation`.

destructive

Argument	Type	Description
datasetId*	string	Dataset id, from caliper_datasets_list.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "datasetId"
  ],
  "properties": {
    "datasetId": {
      "type": "string",
      "description": "Dataset id, from caliper_datasets_list."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    }
  }
}

caliper_datasets_generate

Creates a NEW dataset of model-written items — Q&A by default, or scripted sequences / simulated people via `shape` — pass flowId and the generator reads the flow's prompt, mode, and schema to write realistic cases for THAT flow; `description` adds guidance (or stands alone when there's no flow). Use this when the user wants test cases fast and has none; prefer caliper_datasets_create with hand-written items when real scenarios are already in hand (Compass pages, a transcript). This SPENDS workspace inference credit (one generator call), so it sits behind the approval gate: say so and expect `needs_confirmation`. Returns the dataset summary; read the items with caliper_datasets_get and tell the user to review them before trusting an eval built on them.

Argument	Type	Description
count	integer	Items to generate, 1-50. Default 10.
shape	string	What each generated item is: `qa` (one input, the default), `sequence` (2-6 scripted user turns the model answers one at a time, with expectedResponse + expectedBehavior), or `simulated` (a person Caliper plays adaptively: goal, persona, disposition, expectedBehavior). outputsMode applies to `qa` only.
title*	string	Dataset name.
flowId	string	Workbench flow to generate cases for (from workbench_flows_list). Required unless description is given.
presetId	string	Framing preset. Default `blank`; the others bias every item toward that attack class.
workspace

caliper_datasets_generate_items

Appends model-written items to an EXISTING dataset, in the style of what's already there (existing items are the few-shot examples; the output shape matches theirs unless overridden). Use to widen coverage when the user says 'more like these' or 'add edge cases'; write them by hand with caliper_datasets_add_items when the scenarios are known. SPENDS workspace inference credit, so it sits behind the approval gate — say so. Existing items and their ratings are untouched.

Argument	Type	Description
count	integer	Items to add, 1-20. Default 5.
shape	string	What each generated item is: `qa` (one input, the default), `sequence` (2-6 scripted user turns the model answers one at a time, with expectedResponse + expectedBehavior), or `simulated` (a person Caliper plays adaptively: goal, persona, disposition, expectedBehavior). outputsMode applies to `qa` only.
datasetId*	string	Dataset to extend, from caliper_datasets_list.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.
description	string	What to bias toward, e.g. 'angry customers', 'ambiguous refund questions'.

caliper_datasets_get

One dataset with a page of its items — read this BEFORE editing items (caliper_datasets_update_item needs the item id) and before extending coverage, so you don't add cases that already exist. Items come `limit` at a time (default 25) from `offset`; `total` is the full count. Long fields are cut at ~800 characters with a truncation marker. Get the datasetId from caliper_datasets_list.

read-only

Argument	Type	Description
limit	integer	Items per page, 1-100. Default 25.
offset	integer	Items to skip. Default 0.
datasetId*	string	Dataset id, from caliper_datasets_list.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "datasetId"
  ],
  "properties": {
    "limit": {
      "type": "integer",
      "maximum": 100,
      "minimum": 1,
      "description": "Items per page, 1-100. Default 25."
    },
    "offset": {
      "type": "integer",
      "maximum": 9007199254740991,
      "minimum": 0,
      "description": "Items to skip. Default 0."
    },
    "datasetId": {
      "type": "string",
      "description": "Dataset id, from caliper_datasets_list."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    }
  }
}

caliper_datasets_list

Lists datasets in the active workspace. Returns summaries (id, name, item count); items are not included.

read-only

Argument	Type	Description
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "properties": {
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    }
  }
}

caliper_datasets_remove_item

Drops one item. Ratings and run results keyed to it are orphaned (kept in history, gone from the dataset). A dataset must keep at least one item. May return `needs_confirmation`.

destructive

Argument	Type	Description
itemId*	string	Item id, from caliper_datasets_get.
datasetId*	string	Dataset id.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "datasetId",
    "itemId"
  ],
  "properties": {
    "itemId": {
      "type": "string",
      "description": "Item id, from caliper_datasets_get."
    },
    "datasetId": {
      "type": "string",
      "description": "Dataset id."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    }
  }
}

caliper_datasets_update

Changes a dataset's title, description, or visibility. Pass only what changes; items are untouched (use caliper_datasets_add_items / caliper_datasets_update_item / caliper_datasets_remove_item for those). May return `needs_confirmation`.

destructive

Argument	Type	Description
title	string	New name.
datasetId*	string	Dataset id, from caliper_datasets_list.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.
visibility	string	Who can see it: PRIVATE (only the user), WORKSPACE (every member, the default), or SHARED (specific people, granted afterwards). Say 'make it private' → PRIVATE.
description	any	New description; null clears it.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "datasetId"
  ],
  "properties": {
    "title": {
      "type": "string",
      "maxLength": 120,
      "minLength": 2,
      "description": "New name."
    },
    "datasetId": {
      "type": "string",
      "description": "Dataset id, from caliper_datasets_list."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    },
    "visibility": {
      "enum": [
        "PRIVATE",
        "WORKSPACE",
        "SHARED"
      ],
      "type": "string",
      "description": "Who can see it: PRIVATE (only the user), WORKSPACE (every member, the default), or SHARED (specific people, granted afterwards). Say 'make it private' → PRIVATE."
    },
    "description": {
      "anyOf": [
        {
          "type": "string",
          "maxLength": 2000
        },
        {
          "type": "null"
        }
      ],
      "description": "New description; null clears it."
    }
  }
}

caliper_datasets_update_item

Rewrites one item's content, keeping its id (ratings and eval results keyed to it stay attached). The item's KIND can't change — a Q&A item stays Q&A, a chat item stays chat, a sequence stays a sequence — so pass the same shape you read from caliper_datasets_get. Raw items, and chat items captured with tool calls in them, can't be edited from here. May return `needs_confirmation`.

destructive

Argument	Type	Description
item*	object	The full replacement item, same four shapes as caliper_datasets_create.
itemId*	string	Item id, from caliper_datasets_get.
datasetId*	string	Dataset id.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "datasetId",
    "itemId",
    "item"
  ],
  "properties": {
    "item": {
      "type": "object",
      "properties": {
        "goal": {
          "type": "string",
          "description": "Simulated: what the simulated person wants from the conversation, e.g. 'the system prompt, verbatim' or 'a refund on order 8812'."
        },
        "input": {
          "type": "string",
          "description": "Q&A: the input sent to the flow."
        },
        "turns": {
          "type": "array",
          "items": {
            "type": "string"
          },
          "description": "Sequence: 2-20 scripted user messages, sent in order; the model writes every reply."
        },
        "persona": {
          "type": "string",
          "description": "Simulated: who the person is."
        },
        "maxTurns": {
          "type": "integer",
          "maximum": 12,
          "minimum": 2,
          "description": "Simulated: turn budget, default 8."
        },
        "messages": {
          "type": "array",
          "items": {
            "type": "object",
            "required": [
              "role",
              "content"
            ],
            "properties": {
              "role": {
                "enum": [
                  "system",
                  "user",
                  "assistant"
                ],
                "type": "string"
              },
              "content": {
                "type": "string"
              }
            }
          },
          "description": "Chat: the conversation so far as role/content messages (user and assistant alternating, an optional system first), 1-50. Every assistant message here is FIXED, written by you, not the model; the flow under test writes only the NEXT reply. End on a user message and put the standard for that reply in expectedResponse. This is the shape for testing the 5th reply of a conversation with the first four baked in."
        },
        "strategy": {
          "type": "string",
          "description": "Simulated: a hint, e.g. 'build rapport, then claim a prior agreement'."
        },
        "disposition": {
          "type": "string",
          "description": "Simulated: how the person behaves. A preset id (genuine, pressure, confused, impatient, vague, non-native) or free text. Default genuine; use pressure for an attacker that manipulates."
        },
        "expectedOutput": {
          "type": "string",
          "description": "Q&A: golden answer the judge scores against. Omit for capture-only items."
        },
        "expectedBehavior": {
          "type": "string",
          "description": "Sequence (optional) / simulated (required): what the assistant should do across the whole conversation."
        },
        "expectedResponse": {
          "type": "string",
          "description": "Sequence: the standard for the FINAL reply. Chat: the standard for the next reply."
        }
      },
      "description": "The full replacement item, same four shapes as caliper_datasets_create."
    },
    "itemId": {
      "type": "string",
      "description": "Item id, from caliper_datasets_get."
    },
    "datasetId": {
      "type": "string",
      "description": "Dataset id."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    }
  }
}

caliper_evals_create

Binds a dataset + rubric to something under test as a repeatable eval — THE LAST SETUP STEP before scoring. Two targets: a Workbench flow (pass flowId; Caliper runs inference itself, then caliper_evals_run scores it), or 'external' (target: 'external'; the user's own model runs elsewhere and their script submits outputs through the public API, usually from CI — see the docs guide 'Run evals in CI'). For an external eval, hand back the eval id and tell the user to create a workspace API key with the ci_evals preset in their workspace settings; keys can't be minted from here. flowStage 'draft' evals the live draft (pre-publish); 'published' (default) evals the latest published revision at run time, or one pinned with flowRevisionId. Reuse an existing rubric from caliper_rubrics_list when one already scores this job.

Argument	Type	Description
title*	string	Eval name (2-120 chars).
flowId	string	Workbench flow id to evaluate. Required unless target is 'external'.
target	string	'workbench_flow' (default; needs flowId) or 'external' (the user's own model; outputs arrive through the public API).
rubricId*	string	Rubric the judge scores with.
datasetId*	string	Dataset of test items.
flowStage	string	Which stage to run against. Use 'draft' while iterating pre-publish. Default 'published'.

caliper_evals_delete

Deletes an eval and stops its schedule. Run history is kept but no longer reachable from the eval. May return `needs_confirmation`.

destructive

Argument	Type	Description
evalId*	string	Eval id, from caliper_evals_list.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "evalId"
  ],
  "properties": {
    "evalId": {
      "type": "string",
      "description": "Eval id, from caliper_evals_list."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    }
  }
}

caliper_evals_get

One eval: what it targets (Workbench flow + stage, or external), its dataset and rubric ids, the rubric criteria it scores with (the snapshot taken at creation), schedule + runOnPublish + regression settings, run counts, latest score, and `binding` health (datasetOk / flowOk false = a run would fail at resolution — say so before proposing caliper_evals_run). Get the evalId from caliper_evals_list or caliper_flow_performance.

read-only

Argument	Type	Description
evalId*	string	Eval id.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "evalId"
  ],
  "properties": {
    "evalId": {
      "type": "string",
      "description": "Eval id."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    }
  }
}

caliper_evals_list

Lists evals in the active workspace with their target config (which Workbench flow, which dataset/rubric). Use to find the evalId for caliper_evals_run.

read-only

Argument	Type	Description
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "properties": {
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    }
  }
}

caliper_evals_run

Queues a new run of an eval — inference over the dataset, then LLM-judge scoring. THE VERIFY STEP of the improvement loop: after an approved workbench_flows_edit_text, run the eval again and report the score delta vs the previous run. Runs take a while — but you're brought back into THIS conversation automatically with the scores the moment it finishes, so tell the user it's queued and that you'll follow up here; never poll or ask them to check back. Costs workspace LLM budget, so it sits behind the approval gate: may return `needs_confirmation`.

Argument	Type	Description
label	string	Short label for the run, e.g. 'after refund-policy fix'.
evalId*	string	Eval to run.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string	Approval id from a prior needs_confirmation envelope.
scenarioId	any	Run as this scenario instead of the eval's own (one of the flow's, from workbench_flows_scenarios_list); null runs with none. Omit to use the eval's. Running the same eval as two scenarios compares how the flow does for each person.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "evalId"
  ],
  "properties": {
    "label": {
      "type": "string",
      "description": "Short label for the run, e.g. 'after refund-policy fix'."
    },
    "evalId": {
      "type": "string",
      "description": "Eval to run."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation envelope."
    },
    "scenarioId": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "description": "Run as this scenario instead of the eval's own (one of the flow's, from workbench_flows_scenarios_list); null runs with none. Omit to use the eval's. Running the same eval as two scenarios compares how the flow does for each person."
    }
  }
}

caliper_evals_runs_cancel

Stops a run that is still PENDING / RUNNING / SCORING — the brake on a run that's spending more than expected or was started by mistake. The queue stops at once; an item already handed to the flow finishes on its own timeout. The run settles as FAILED with a 'cancelled' reason and keeps the items it completed. A run that already finished returns run_not_live. May return `needs_confirmation`.

destructive

Argument	Type	Description
runId*	string	Run id, from caliper_evals_run or caliper_evals_runs_list.
evalId*	string	Eval the run belongs to.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "evalId",
    "runId"
  ],
  "properties": {
    "runId": {
      "type": "string",
      "description": "Run id, from caliper_evals_run or caliper_evals_runs_list."
    },
    "evalId": {
      "type": "string",
      "description": "Eval the run belongs to."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    }
  }
}

caliper_evals_runs_get

One run: status, overall score, and by default the ten lowest-scoring items with their per-criterion scores and the judge's reasoning (`items: all` for every item, `none` for totals only). THE ANSWER to 'why did the score drop' — read this, then name the criterion that slipped and quote the reasoning on the lowest items. Works for runs Caliper ran and for runs submitted from CI (triggeredBy 'ci', usually labeled with a pull request number). Still no polling loops; read it when the user asks.

read-only

Argument	Type	Description
items	string	Which items to include. `worst` (default) = the lowest-scoring `limit` items, the ones that explain a drop; `all` = every item (large); `none` = the run's totals only.
limit	integer	How many items for `worst`, default 10.
runId*	string	Run id, from caliper_evals_runs_list or caliper_evals_run.
evalId*	string	Eval the run belongs to.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "evalId",
    "runId"
  ],
  "properties": {
    "items": {
      "enum": [
        "worst",
        "all",
        "none"
      ],
      "type": "string",
      "description": "Which items to include. `worst` (default) = the lowest-scoring `limit` items, the ones that explain a drop; `all` = every item (large); `none` = the run's totals only."
    },
    "limit": {
      "type": "integer",
      "maximum": 100,
      "minimum": 1,
      "description": "How many items for `worst`, default 10."
    },
    "runId": {
      "type": "string",
      "description": "Run id, from caliper_evals_runs_list or caliper_evals_run."
    },
    "evalId": {
      "type": "string",
      "description": "Eval the run belongs to."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    }
  }
}

caliper_evals_runs_item_execution

What the flow actually DID on one run item: every step in order (what it said, which tools it called with what arguments, what came back) and the final answer, plus status, duration, and cost. Read this when a low score needs explaining beyond the judge's reasoning — a wrong tool call or an empty tool result is usually the cause, and the fix is different from a prompt fix. Pass the run item's `id` (NOT `itemId`) from caliper_evals_runs_get. Items whose outputs were supplied from outside (external evals) have no trace and return not_found. Steps are capped for transport; the run page in Caliper has the full record.

read-only

Argument	Type	Description
runId*	string	Run id.
evalId*	string	Eval the run belongs to.
runItemId*	string	The run item's `id` from caliper_evals_runs_get (not its dataset itemId).
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "evalId",
    "runId",
    "runItemId"
  ],
  "properties": {
    "runId": {
      "type": "string",
      "description": "Run id."
    },
    "evalId": {
      "type": "string",
      "description": "Eval the run belongs to."
    },
    "runItemId": {
      "type": "string",
      "description": "The run item's `id` from caliper_evals_runs_get (not its dataset itemId)."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    }
  }
}

caliper_evals_runs_list

Recent runs for one eval, newest first: status (PENDING/RUNNING/SCORING/DONE/FAILED), overall score once DONE, label, and timestamps. THE CHECK-BACK for caliper_evals_run: when the user asks how the run went, read this — cite the run id and score, and compare against the PREVIOUS run's score for the delta. Still don't poll in a loop; check when the user asks or when reporting.

read-only

Argument	Type	Description
evalId*	string	Eval whose runs to list.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "evalId"
  ],
  "properties": {
    "evalId": {
      "type": "string",
      "description": "Eval whose runs to list."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    }
  }
}

caliper_evals_update

Changes an eval's title, description, visibility, schedule (interval + time anchor), runOnPublish (queue a run whenever the flow publishes), regressionThreshold (score drop vs the previous run that triggers an alert; null = off), or scenarioId (the scenario runs use). Pass only what changes. The dataset, rubric snapshot, and target are fixed at creation — create a new eval to change those. Scheduled and on-publish runs spend credit on their own, so state that plainly when turning them on. May return `needs_confirmation`.

destructive

Argument	Type	Description
title	string	New name.
evalId*	string	Eval id, from caliper_evals_list.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.
scenarioId	any	Workbench-flow evals: the scenario every run uses unless a run picks another (one of the flow's, from workbench_flows_scenarios_list); null clears it.
visibility	string	Who can see it: PRIVATE (only the user), WORKSPACE (every member, the default), or SHARED (specific people, granted afterwards). Say 'make it private' → PRIVATE.

caliper_flow_performance

How a Workbench flow is ACTUALLY doing, with receipts: every Caliper eval targeting the flow, recent runs with scores, the latest run decomposed into per-criterion averages, the score delta vs the previous run, and the worst-scoring items WITH the judge's reasoning. Use this BEFORE claiming a flow works or proposing changes — and cite the runId + scores when you do. The worst items are diagnostic: failures clustered around missing company facts suggest a knowledge gap (consider proposing a Compass interview with the workflow owner) rather than a prompt problem.

read-only

Argument	Type	Description
flowId*	string	Workbench flow id to report on.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "flowId"
  ],
  "properties": {
    "flowId": {
      "type": "string",
      "description": "Workbench flow id to report on."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    }
  }
}

caliper_rubrics_create

Creates the scoring rubric an eval's LLM judge uses — 1-10 criteria, each scored on a numeric scale (default 1-5), judged pass/fail (kind pass_fail), or checked in code with no judge (kind check: contains, not_contains, matches a regex, valid_json, max_chars, equals_expected). Write criteria about the FLOW'S JOB (accuracy to source material, tone, refusal behavior), not generic 'quality'.

Argument	Type	Description
title*	string	Rubric name (2-120 chars).
criteria*	array
workspace	string	Workspace slug override.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.
visibility	string	Who can see it: PRIVATE (only the user), WORKSPACE (every member, the default), or SHARED (specific people, granted afterwards). Say 'make it private' → PRIVATE.
description	string

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "title",
    "criteria"
  ],
  "properties": {
    "title": {
      "type": "string",
      "description": "Rubric name (2-120 chars)."
    },
    "criteria": {
      "type": "array",
      "items": {
        "type": "object",
        "required": [
          "name"
        ],
        "properties": {
          "kind": {
            "enum": [
              "scale",
              "pass_fail",
              "check"
            ],
            "type": "string",
            "description": "scale (default): judged on scaleMin..scaleMax. pass_fail: judged pass or fail. check: decided in code with no judge — set `check`."
          },
          "name": {
            "type": "string",
            "description": "Criterion name (2-80 chars)."
          },
          "check": {
            "type": "object",
            "required": [
              "type"
            ],
            "properties": {
              "type": {
                "enum": [
                  "contains",
                  "not_contains",
                  "matches",
                  "valid_json",
                  "max_chars",
                  "equals_expected"
                ],
                "type": "string"
              },
              "value": {
                "type": "string",
                "description": "Text for contains/not_contains, a regex for matches, a number for max_chars."
              },
              "caseSensitive": {
                "type": "boolean"
              }
            },
            "description": "For kind=check: what the code looks for in the final reply."
          },
          "scaleMax": {
            "type": "integer",
            "maximum": 9007199254740991,
            "minimum": -9007199254740991,
            "description": "Default 5."
          },
          "scaleMin": {
            "type": "integer",
            "maximum": 9007199254740991,
            "minimum": -9007199254740991,
            "description": "Default 1. Ignored for pass_fail and check (always 0..1)."
          },
          "description": {
            "type": "string",
            "description": "What the judge should look for (max 500 chars)."
          }
        }
      },
      "maxItems": 10,
      "minItems": 1
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug override."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    },
    "visibility": {
      "enum": [
        "PRIVATE",
        "WORKSPACE",
        "SHARED"
      ],
      "type": "string",
      "description": "Who can see it: PRIVATE (only the user), WORKSPACE (every member, the default), or SHARED (specific people, granted afterwards). Say 'make it private' → PRIVATE."
    },
    "description": {
      "type": "string"
    }
  }
}

caliper_rubrics_delete

Deletes a rubric. Existing evals keep their snapshot of it and keep running; nothing new can bind to it. May return `needs_confirmation`.

destructive

Argument	Type	Description
rubricId*	string	Rubric id, from caliper_rubrics_list.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "rubricId"
  ],
  "properties": {
    "rubricId": {
      "type": "string",
      "description": "Rubric id, from caliper_rubrics_list."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    }
  }
}

caliper_rubrics_get

One rubric with its criteria — each criterion's name, what the judge looks for, and the score scale. Read this to explain a score (which criterion slipped and what it asks for) or before caliper_rubrics_update. Get the rubricId from caliper_rubrics_list or an eval's rubricId.

read-only

Argument	Type	Description
rubricId*	string	Rubric id.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "rubricId"
  ],
  "properties": {
    "rubricId": {
      "type": "string",
      "description": "Rubric id."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    }
  }
}

caliper_rubrics_list

Lists the rubrics in the active workspace (id, name, description, criterion count). Check here BEFORE caliper_rubrics_create — reuse an existing rubric's id in caliper_evals_create when one already scores the same job. Read a rubric's criteria with caliper_rubrics_get.

read-only

Argument	Type	Description
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "properties": {
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    }
  }
}

caliper_rubrics_update

Changes a rubric's title, description, visibility, or replaces its criteria wholesale (pass the FULL list — criteria get fresh ids). Evals snapshot the rubric when they're created, so an existing eval keeps scoring with the criteria it started with; say so, and offer to create a new eval when the criteria change materially. May return `needs_confirmation`.

destructive

Argument	Type	Description
title	string	New name.
criteria	array	Replacement criteria (1-10) — the whole list, not a diff.
rubricId*	string	Rubric id, from caliper_rubrics_list.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.
visibility	string	Who can see it: PRIVATE (only the user), WORKSPACE (every member, the default), or SHARED (specific people, granted afterwards). Say 'make it private' → PRIVATE.
description	any	New description; null clears it.

caliper_source_feeds_list

Datasets this source keeps adding to as conversations arrive: which dataset, for what (review, eval, spec), the pick it matches, sampling, how many have landed, and whether it's still running or why it stopped.

read-only

Argument	Type	Description
sourceId*	string	Source id, from caliper_sources_list or search_workspace.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "sourceId"
  ],
  "properties": {
    "sourceId": {
      "type": "string",
      "description": "Source id, from caliper_sources_list or search_workspace."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    }
  }
}

caliper_source_feeds_stop

Stops a source from adding new conversations to a dataset. What it already added stays. Get the feedId from caliper_source_feeds_list. May return `needs_confirmation`.

destructive

Argument	Type	Description
feedId*	string	Live dataset id, from caliper_source_feeds_list.
sourceId*	string	Source id, from caliper_sources_list or search_workspace.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "sourceId",
    "feedId"
  ],
  "properties": {
    "feedId": {
      "type": "string",
      "description": "Live dataset id, from caliper_source_feeds_list."
    },
    "sourceId": {
      "type": "string",
      "description": "Source id, from caliper_sources_list or search_workspace."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    }
  }
}

caliper_sources_list

Apps sending their agent's traces to Caliper — from their own code, OpenTelemetry, or a published Workbench flow — busiest first: name, how it sends, traces in the last 14 days (per day), and the tools its agent calls most with the share of traces using each. Start here when the user asks how their agent behaves on real traffic; then caliper_sources_over_time for one source.

read-only

Argument	Type	Description
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "properties": {
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    }
  }
}

caliper_sources_over_time

One source day by day: traces, failures, tool calls, lookups that found nothing, tokens, cost, and speed (meanMs; p50/p95 as 'answered within' bucket edges); each tool with its calls and the share of each day's traces that used it; the documents lookups landed on most; models used. Use it to answer 'how often is it calling web search and is that changing' — compare the first and last weeks and name the day it moved. Follow with caliper_traces_list on that day or tool to show the conversations behind it.

read-only

Argument	Type	Description
days	integer	How many days back, default 30.
sourceId*	string	Source id, from caliper_sources_list or search_workspace.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "sourceId"
  ],
  "properties": {
    "days": {
      "type": "integer",
      "maximum": 365,
      "minimum": 1,
      "description": "How many days back, default 30."
    },
    "sourceId": {
      "type": "string",
      "description": "Source id, from caliper_sources_list or search_workspace."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    }
  }
}

caliper_sources_to_dataset

Saves the conversations behind a pick (a day, a tool, a document, failures — the same narrowing as caliper_traces_list) into a new dataset (newDatasetName) or an existing one (datasetId), newest first up to `limit`. `purpose` decides what each item keeps: review = the agent's reply, tool calls included, for people to rate; eval = the reply becomes the expected answer; spec = only the questions, for people to answer. Set keepAdding to make it live: new matching conversations keep arriving (every one, or 1 in 10 / 1 in 100), up to 1,000. Then offer the next step — caliper_evals_create, or a review in Caliper. May return `needs_confirmation`.

Argument	Type	Description
day	string	One UTC day, YYYY-MM-DD.
tool	string	Only traces that called this tool.
limit	integer	Most recent N, default 100.
purpose*	string	review \| eval \| spec.
document	string	Only traces whose lookups hit this document.
sourceId*	string	Source id, from caliper_sources_list or search_workspace.
datasetId	string	Add to this dataset…
workspace	string

caliper_starters_install

Copies one starter into the workspace as a new dataset and a new rubric the user owns and can edit. Returns both ids; the next step is caliper_evals_create binding them to the flow (or target 'external'). Needs caliper:datasets:write and caliper:evals:write.

Argument	Type	Description
slug*	string	Starter slug from caliper_starters_list.
workspace	string	Workspace slug override.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "slug"
  ],
  "properties": {
    "slug": {
      "enum": [
        "prompt-injection",
        "system-prompt-leaking",
        "over-refusal",
        "personal-data",
        "bias-under-ambiguity"
      ],
      "type": "string",
      "description": "Starter slug from caliper_starters_list."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug override."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    }
  }
}

caliper_starters_list

The starter sets Caliper ships: prompt injection, system-prompt leaking, over-refusal, personal-data handling, bias under ambiguity. Each is a small original dataset (single messages AND multi-turn build-ups: false memory, fabricated earlier turns, slow escalation) paired with an anchored rubric. Suggest one when a user wants to check an assistant for these and has no cases yet; install with caliper_starters_install. Say plainly that a starter is a smoke test (the items are public), not a safety score — their own cases are where the real signal is.

read-only

No arguments.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "properties": {}
}

caliper_traces_get

One trace: what came in, what went out, and every step in order — model calls (model, tokens, cost, reasoning), tool calls (arguments and result), lookups (query and documents with scores), anything else — each with timing and status. Long values are cut at 2,000 characters. Use it to explain why one conversation went the way it did.

read-only

Argument	Type	Description
traceId*	string	Trace id, from caliper_traces_list.
sourceId*	string	Source id, from caliper_sources_list or search_workspace.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "sourceId",
    "traceId"
  ],
  "properties": {
    "traceId": {
      "type": "string",
      "description": "Trace id, from caliper_traces_list."
    },
    "sourceId": {
      "type": "string",
      "description": "Source id, from caliper_sources_list or search_workspace."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    }
  }
}

caliper_traces_list

The traces behind a point on a source's chart, newest first, 50 a page: the question and answer (cut short), tools called, time, tokens, cost, status. Narrow by day, tool, document, or failures; page with `before` (the nextBefore from the last page). Open one with caliper_traces_get.

read-only

Argument	Type	Description
day	string	One UTC day, YYYY-MM-DD.
tool	string	Only traces that called this tool.
before	string	Paging: the nextBefore from the previous page.
document	string	Only traces whose lookups hit this document.
sourceId*	string	Source id, from caliper_sources_list or search_workspace.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
errorsOnly	boolean	Only failed traces.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "sourceId"
  ],
  "properties": {
    "day": {
      "type": "string",
      "pattern": "^\\d{4}-\\d{2}-\\d{2}$",
      "description": "One UTC day, YYYY-MM-DD."
    },
    "tool": {
      "type": "string",
      "description": "Only traces that called this tool."
    },
    "before": {
      "type": "string",
      "description": "Paging: the nextBefore from the previous page."
    },
    "document": {
      "type": "string",
      "description": "Only traces whose lookups hit this document."
    },
    "sourceId": {
      "type": "string",
      "description": "Source id, from caliper_sources_list or search_workspace."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "errorsOnly": {
      "type": "boolean",
      "description": "Only failed traces."
    }
  }
}

comments_create

Posts a comment on a workspace entity — a new thread, or a reply when rootId is given. Use it to leave findings where the discussion already lives (an eval result on the flow being debated, a summary on a long thread). Mention people via mentionedUserIds (from workspace member ids) to ring their notification bell; never mention someone who didn't ask to be pulled in.

Argument	Type	Description
body*	string
rootId	string	Reply into this thread; omit to start a new one.
entityId*	string
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string
entityKind*	string	What the thread hangs on.
mentionedUserIds	array

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "entityKind",
    "entityId",
    "body"
  ],
  "properties": {
    "body": {
      "type": "string",
      "maxLength": 10000,
      "minLength": 1
    },
    "rootId": {
      "type": "string",
      "description": "Reply into this thread; omit to start a new one."
    },
    "entityId": {
      "type": "string",
      "maxLength": 60,
      "minLength": 1
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string"
    },
    "entityKind": {
      "enum": [
        "workbench_flow",
        "compass_page",
        "compass_opportunity",
        "caliper_review",
        "caliper_dataset",
        "caliper_rubric",
        "caliper_eval",
        "caliper_spec",
        "napkin_board",
        "napkin_diagram",
        "napkin_doc",
        "napkin_sheet",
        "napkin_interface",
        "ledger_entry",
        "prism_field",
        "workspace_file"
      ],
      "type": "string",
      "description": "What the thread hangs on."
    },
    "mentionedUserIds": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "maxItems": 20
    }
  }
}

comments_list

Lists the comment threads on one workspace entity (open first, then resolved) with authors and timestamps. Read this before weighing in on contested work — the threads are where disagreement lives before it becomes a decision.

read-only

Argument	Type	Description
entityId*	string
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
entityKind*	string	What the thread hangs on.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "entityKind",
    "entityId"
  ],
  "properties": {
    "entityId": {
      "type": "string",
      "maxLength": 60,
      "minLength": 1
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "entityKind": {
      "enum": [
        "workbench_flow",
        "compass_page",
        "compass_opportunity",
        "caliper_review",
        "caliper_dataset",
        "caliper_rubric",
        "caliper_eval",
        "caliper_spec",
        "napkin_board",
        "napkin_diagram",
        "napkin_doc",
        "napkin_sheet",
        "napkin_interface",
        "ledger_entry",
        "prism_field",
        "workspace_file"
      ],
      "type": "string",
      "description": "What the thread hangs on."
    }
  }
}

comments_resolve

Sets a comment thread's resolved state (rootId = the thread's root comment id). Resolve ONLY when the human asked or the thread's question is demonstrably settled — and say what settled it in a reply first. Reopening is for new evidence.

destructive

Argument	Type	Description
rootId*	string
resolved*	boolean
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "rootId",
    "resolved"
  ],
  "properties": {
    "rootId": {
      "type": "string",
      "maxLength": 60,
      "minLength": 1
    },
    "resolved": {
      "type": "boolean"
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string"
    }
  }
}

entity_tags_browse

Without a tag: every tag in use across the workspace with how many entities carry it, most-used first — the vocabulary the team already organizes by. With a tag: everything filed under it across every tool, each with its kind, id, title, and path. Use it to reuse existing labels instead of inventing near-duplicates, and to answer 'show me everything about X' when X is a label.

read-only

Argument	Type	Description
tag	string	A tag to expand into its items. Omit to list tags.
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "properties": {
    "tag": {
      "type": "string",
      "maxLength": 40,
      "minLength": 1,
      "description": "A tag to expand into its items. Omit to list tags."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    }
  }
}

entity_tags_get

Returns the tags on a batch of entities of one kind — the labels galleries organize by. Ids come from the kind's list/get tool or from search_workspace. Use it before entity_tags_set so you replace the full set knowingly, and to answer 'what is this filed under'. Entities the user can't see are omitted.

read-only

Argument	Type	Description
ids*	array
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
entityKind*	string	Which kind the ids belong to.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "entityKind",
    "ids"
  ],
  "properties": {
    "ids": {
      "type": "array",
      "items": {
        "type": "string",
        "maxLength": 100,
        "minLength": 1
      },
      "maxItems": 100,
      "minItems": 1
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "entityKind": {
      "enum": [
        "workbench_flow",
        "workbench_task",
        "workbench_kb",
        "compass_page",
        "compass_opportunity",
        "caliper_dataset",
        "caliper_rubric",
        "caliper_eval",
        "caliper_review",
        "caliper_spec",
        "caliper_source",
        "ledger_entry",
        "ledger_metric",
        "napkin_board",
        "napkin_deck",
        "napkin_doc",
        "napkin_sheet",
        "napkin_diagram",
        "prism_field",
        "prism_study"
      ],
      "type": "string",
      "description": "Which kind the ids belong to."
    }
  }
}

entity_tags_set

Replaces the FULL tag set on one entity (an empty list clears it). Read the current tags with entity_tags_get first and pass the merged list — this is not additive. Tags are lowercase letters, numbers, spaces, and hyphens; prefer labels already in use (entity_tags_browse) so the workspace's vocabulary stays small. The id comes from the kind's list/get tool or search_workspace.

destructive

Argument	Type	Description
tags*	array
entityId*	string
workspace	string	Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalId	string	Approval id from a prior needs_confirmation response. Omit on the first call.
entityKind*	string

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "entityKind",
    "entityId",
    "tags"
  ],
  "properties": {
    "tags": {
      "type": "array",
      "items": {
        "type": "string",
        "pattern": "^[a-z0-9](?:[a-z0-9 -]*[a-z0-9])?$",
        "maxLength": 40
      },
      "maxItems": 50
    },
    "entityId": {
      "type": "string",
      "maxLength": 100,
      "minLength": 1
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    },
    "entityKind": {
      "enum": [
        "workbench_flow",
        "workbench_task",
        "workbench_kb",
        "compass_page",
        "compass_opportunity",
        "caliper_dataset",
        "caliper_rubric",
        "caliper_eval",
        "caliper_review",
        "caliper_spec",
        "caliper_source",
        "ledger_entry",
        "ledger_metric",
        "napkin_board",
        "napkin_deck",
        "napkin_doc",
        "napkin_sheet",
        "napkin_diagram",
        "prism_field",
        "prism_study"
      ],
      "type": "string"
    }
  }
}

get_doc

Fetch the full Markdown body of a specific docs page by its slug. Use this after `search_docs` when the user needs the complete content of a page. No authentication required.

read-only

Argument	Type	Description
slug*	string	Page slug. Accepts 'compass/api', '/compass/api', or 'docs/compass/api'.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "slug"
  ],
  "properties": {
    "slug": {
      "type": "string",
      "minLength": 1,
      "description": "Page slug. Accepts 'compass/api', '/compass/api', or 'docs/compass/api'."
    }
  }
}

list_docs

Enumerate all available docs pages, optionally filtered by product (e.g. 'compass', 'legal', 'overview'). Use this to discover what slugs exist before calling `get_doc`. No authentication required.

read-only

Argument	Type	Description
product	string	Optional product slug filter (e.g. 'compass', 'legal', 'overview').

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "properties": {
    "product": {
      "type": "string",
      "description": "Optional product slug filter (e.g. 'compass', 'legal', 'overview')."
    }
  }
}

search_docs

Search ZeroWidth product documentation. Returns matching pages with title, slug, public URL, and a query-relevant snippet. Use this when the user asks about a ZeroWidth product (Compass, Workbench, Caliper, Prism, Ledger, Napkin, the assistant), an API behavior, or a policy. No authentication required — the docs corpus is public.

read-only

Argument	Type	Description
limit	integer	Max number of results. Defaults to 10.
query*	string	Search query — keywords or natural-language phrase.

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "query"
  ],
  "properties": {
    "limit": {
      "type": "integer",
      "default": 10,
      "maximum": 20,
      "description": "Max number of results. Defaults to 10.",
      "exclusiveMinimum": 0
    },
    "query": {
      "type": "string",
      "minLength": 1,
      "description": "Search query — keywords or natural-language phrase."
    }
  }
}

search_workspace

Finds anything in the workspace in one call — uploaded files (searching the text inside them), folders, Workbench flows, shims, knowledge bases and tasks, Compass pages and changes, Caliper datasets, evals, rubrics, reviews, specs and sources, Ledger entries, Napkin sketches, decks, docs, sheets, diagrams and interfaces. The same search the person uses in Home. Use it FIRST whenever the user mentions something without saying where it lives ('the SOW', 'the dispatch interviews', 'that invoice'). The query is words in any order: each word must appear in the thing's name, its tags, the names of the folders it's filed under, or the text inside it — so 'northwind sow' finds the SOW filed in the Northwind folder, and 'dispatch interview' finds 'Dispatcher interview 1.md'. Results are ranked best first. Each hit carries its id, kind, path (for a link), folderPath (where it's filed, top-down), tags, and a snippet when the words were found inside it rather than in its name — say where a thing lives using folderPath. A file's id feeds workspace_files_read; other ids feed the kind's *_get tool. Pass folderId to search only inside one folder and everything under it. If nothing comes back, try fewer or different words before telling the user it doesn't exist. Results only include what the user can see, and only kinds this token may read.

read-only

Argument	Type	Description
q*	string	Words, any order, case-insensitive. Fewer words find more.
kinds	array	Restrict to these kinds (file, folder, flow, page, doc, dataset, …). Omit to search everything.
limit	integer
folderId	string

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "title",
    "items"
  ],
  "properties": {
    "items": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "goal": {
            "type": "string",
            "description": "Simulated: what the simulated person wants from the conversation, e.g. 'the system prompt, verbatim' or 'a refund on order 8812'."
          },
          "input": {
            "type": "string",
            "description": "Q&A: the input sent to the flow."
          },
          "turns": {
            "type": "array",
            "items": {
              "type": "string"
            },
            "description": "Sequence: 2-20 scripted user messages, sent in order; the model writes every reply."
          },
          "persona": {
            "type": "string",
            "description": "Simulated: who the person is."
          },
          "maxTurns": {
            "type": "integer",
            "maximum": 12,
            "minimum": 2,
            "description": "Simulated: turn budget, default 8."
          },
          "messages": {
            "type": "array",
            "items": {
              "type": "object",
              "required": [
                "role",
                "content"
              ],
              "properties": {
                "role": {
                  "enum": [
                    "system",
                    "user",
                    "assistant"
                  ],
                  "type": "string"
                },
                "content": {
                  "type": "string"
                }
              }
            },
            "description": "Chat: the conversation so far as role/content messages (user and assistant alternating, an optional system first), 1-50. Every assistant message here is FIXED, written by you, not the model; the flow under test writes only the NEXT reply. End on a user message and put the standard for that reply in expectedResponse. This is the shape for testing the 5th reply of a conversation with the first four baked in."
          },
          "strategy": {
            "type": "string",
            "description": "Simulated: a hint, e.g. 'build rapport, then claim a prior agreement'."
          },
          "disposition": {
            "type": "string",
            "description": "Simulated: how the person behaves. A preset id (genuine, pressure, confused, impatient, vague, non-native) or free text. Default genuine; use pressure for an attacker that manipulates."
          },
          "expectedOutput": {
            "type": "string",
            "description": "Q&A: golden answer the judge scores against. Omit for capture-only items."
          },
          "expectedBehavior": {
            "type": "string",
            "description": "Sequence (optional) / simulated (required): what the assistant should do across the whole conversation."
          },
          "expectedResponse": {
            "type": "string",
            "description": "Sequence: the standard for the FINAL reply. Chat: the standard for the next reply."
          }
        }
      },
      "maxItems": 100,
      "minItems": 1,
      "description": "Test items (1-100)."
    },
    "title": {
      "type": "string",
      "description": "Dataset name (2-120 chars)."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    },
    "visibility": {
      "enum": [
        "PRIVATE",
        "WORKSPACE",
        "SHARED"
      ],
      "type": "string",
      "description": "Who can see it: PRIVATE (only the user), WORKSPACE (every member, the default), or SHARED (specific people, granted afterwards). Say 'make it private' → PRIVATE."
    },
    "description": {
      "type": "string",
      "description": "What this dataset covers."
    }
  }
}

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "title"
  ],
  "properties": {
    "count": {
      "type": "integer",
      "maximum": 50,
      "minimum": 1,
      "description": "Items to generate, 1-50. Default 10."
    },
    "shape": {
      "enum": [
        "qa",
        "sequence",
        "simulated"
      ],
      "type": "string",
      "description": "What each generated item is: `qa` (one input, the default), `sequence` (2-6 scripted user turns the model answers one at a time, with expectedResponse + expectedBehavior), or `simulated` (a person Caliper plays adaptively: goal, persona, disposition, expectedBehavior). outputsMode applies to `qa` only."
    },
    "title": {
      "type": "string",
      "maxLength": 120,
      "minLength": 2,
      "description": "Dataset name."
    },
    "flowId": {
      "type": "string",
      "description": "Workbench flow to generate cases for (from workbench_flows_list). Required unless description is given."
    },
    "presetId": {
      "enum": [
        "blank",
        "adversarial",
        "prompt-injection",
        "jailbreak",
        "multilingual"
      ],
      "type": "string",
      "description": "Framing preset. Default `blank`; the others bias every item toward that attack class."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    },
    "visibility": {
      "enum": [
        "PRIVATE",
        "WORKSPACE",
        "SHARED"
      ],
      "type": "string",
      "description": "Who can see it: PRIVATE (only the user), WORKSPACE (every member, the default), or SHARED (specific people, granted afterwards). Say 'make it private' → PRIVATE."
    },
    "description": {
      "type": "string",
      "maxLength": 4000,
      "description": "What the target system does / what to cover. Required (≥10 chars) when flowId is omitted; optional guidance otherwise."
    },
    "disposition": {
      "type": "string",
      "maxLength": 2000,
      "description": "With shape `simulated`: how every generated person behaves — a preset id (genuine, pressure, confused, impatient, vague, non-native) or free text. Omit to let the generator vary it from person to person."
    },
    "outputsMode": {
      "enum": [
        "none",
        "expected",
        "captured"
      ],
      "type": "string",
      "description": "What each generated item carries beyond the input: `expected` (golden answers — eval-ready, the default), `none` (inputs only — for a spec others fill in), `captured` (sample answers to rate in a review)."
    },
    "expectedStyle": {
      "enum": [
        "verbatim",
        "conditions"
      ],
      "type": "string",
      "description": "With outputsMode `expected`: `verbatim` literal reference answers (default) or `conditions` — what a correct answer must do, when there's no single right wording."
    },
    "datasetDescription": {
      "type": "string",
      "maxLength": 2000,
      "description": "Description stored on the dataset."
    }
  }
}

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "datasetId"
  ],
  "properties": {
    "count": {
      "type": "integer",
      "maximum": 20,
      "minimum": 1,
      "description": "Items to add, 1-20. Default 5."
    },
    "shape": {
      "enum": [
        "qa",
        "sequence",
        "simulated"
      ],
      "type": "string",
      "description": "What each generated item is: `qa` (one input, the default), `sequence` (2-6 scripted user turns the model answers one at a time, with expectedResponse + expectedBehavior), or `simulated` (a person Caliper plays adaptively: goal, persona, disposition, expectedBehavior). outputsMode applies to `qa` only."
    },
    "datasetId": {
      "type": "string",
      "description": "Dataset to extend, from caliper_datasets_list."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    },
    "description": {
      "type": "string",
      "maxLength": 2000,
      "description": "What to bias toward, e.g. 'angry customers', 'ambiguous refund questions'."
    },
    "disposition": {
      "type": "string",
      "maxLength": 2000,
      "description": "With shape `simulated`: how every generated person behaves — a preset id (genuine, pressure, confused, impatient, vague, non-native) or free text. Omit to let the generator vary it from person to person."
    },
    "outputsMode": {
      "enum": [
        "none",
        "expected",
        "captured"
      ],
      "type": "string",
      "description": "What each generated item carries beyond the input: `expected` (golden answers — eval-ready, the default), `none` (inputs only — for a spec others fill in), `captured` (sample answers to rate in a review)."
    },
    "anchorItemId": {
      "type": "string",
      "description": "An item id (from caliper_datasets_get) the new items should resemble most."
    },
    "expectedStyle": {
      "enum": [
        "verbatim",
        "conditions"
      ],
      "type": "string",
      "description": "With outputsMode `expected`: `verbatim` literal reference answers (default) or `conditions` — what a correct answer must do, when there's no single right wording."
    }
  }
}

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "title",
    "datasetId",
    "rubricId"
  ],
  "properties": {
    "title": {
      "type": "string",
      "description": "Eval name (2-120 chars)."
    },
    "flowId": {
      "type": "string",
      "description": "Workbench flow id to evaluate. Required unless target is 'external'."
    },
    "target": {
      "enum": [
        "workbench_flow",
        "external"
      ],
      "type": "string",
      "description": "'workbench_flow' (default; needs flowId) or 'external' (the user's own model; outputs arrive through the public API)."
    },
    "rubricId": {
      "type": "string",
      "description": "Rubric the judge scores with."
    },
    "datasetId": {
      "type": "string",
      "description": "Dataset of test items."
    },
    "flowStage": {
      "enum": [
        "draft",
        "published"
      ],
      "type": "string",
      "description": "Which stage to run against. Use 'draft' while iterating pre-publish. Default 'published'."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug override."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    },
    "scenarioId": {
      "type": "string",
      "description": "Workbench-flow evals: run every item as this scenario (one of the flow's, from workbench_flows_scenarios_list), so the flow's variables hold that person's values. A run can pick another with caliper_evals_run's scenarioId. Without one, variables run empty unless an item sets them."
    },
    "visibility": {
      "enum": [
        "PRIVATE",
        "WORKSPACE",
        "SHARED"
      ],
      "type": "string",
      "description": "Who can see it: PRIVATE (only the user), WORKSPACE (every member, the default), or SHARED (specific people, granted afterwards). Say 'make it private' → PRIVATE."
    },
    "description": {
      "type": "string"
    },
    "externalLabel": {
      "type": "string",
      "maxLength": 120,
      "description": "External evals only: what produces the outputs, e.g. 'CI' or 'prod pipeline'. Free text, shown on the eval."
    },
    "flowRevisionId": {
      "type": "string",
      "description": "Published-stage evals only: pin to one published revision (id from workbench_flows_revisions_list). Omit to always eval the latest published version at run time."
    }
  }
}

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "evalId"
  ],
  "properties": {
    "title": {
      "type": "string",
      "maxLength": 120,
      "minLength": 2,
      "description": "New name."
    },
    "evalId": {
      "type": "string",
      "description": "Eval id, from caliper_evals_list."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    },
    "scenarioId": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "description": "Workbench-flow evals: the scenario every run uses unless a run picks another (one of the flow's, from workbench_flows_scenarios_list); null clears it."
    },
    "visibility": {
      "enum": [
        "PRIVATE",
        "WORKSPACE",
        "SHARED"
      ],
      "type": "string",
      "description": "Who can see it: PRIVATE (only the user), WORKSPACE (every member, the default), or SHARED (specific people, granted afterwards). Say 'make it private' → PRIVATE."
    },
    "description": {
      "anyOf": [
        {
          "type": "string",
          "maxLength": 2000
        },
        {
          "type": "null"
        }
      ],
      "description": "New description; null clears it."
    },
    "runOnPublish": {
      "type": "boolean",
      "description": "Run whenever the targeted flow publishes a version."
    },
    "scheduleTime": {
      "anyOf": [
        {
          "type": "string",
          "pattern": "^([01]\\d|2[0-3]):[0-5]\\d$"
        },
        {
          "type": "null"
        }
      ],
      "description": "Daily/weekly anchor, HH:MM 24-hour in scheduleTimezone. Null = one interval from now."
    },
    "scheduleInterval": {
      "anyOf": [
        {
          "enum": [
            "hourly",
            "daily",
            "weekly"
          ],
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "description": "Run cadence; null clears the schedule. Each scheduled run spends credit — say so."
    },
    "scheduleTimezone": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "description": "IANA timezone for scheduleTime, e.g. America/Chicago."
    },
    "scheduleDayOfWeek": {
      "anyOf": [
        {
          "type": "integer",
          "maximum": 6,
          "minimum": 0
        },
        {
          "type": "null"
        }
      ],
      "description": "Weekly only: 0 = Sunday … 6 = Saturday."
    },
    "regressionThreshold": {
      "anyOf": [
        {
          "type": "number",
          "maximum": 1,
          "minimum": 0.01
        },
        {
          "type": "null"
        }
      ],
      "description": "Score drop vs the previous run that raises an alert, on the 0–1 score scale (0.1 = 10 points); null turns alerts off."
    }
  }
}

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "rubricId"
  ],
  "properties": {
    "title": {
      "type": "string",
      "maxLength": 120,
      "minLength": 2,
      "description": "New name."
    },
    "criteria": {
      "type": "array",
      "items": {
        "type": "object",
        "required": [
          "name"
        ],
        "properties": {
          "kind": {
            "enum": [
              "scale",
              "pass_fail",
              "check"
            ],
            "type": "string",
            "description": "scale (default): judged on scaleMin..scaleMax. pass_fail: judged pass or fail. check: decided in code with no judge — set `check`."
          },
          "name": {
            "type": "string",
            "description": "Criterion name (2-80 chars)."
          },
          "check": {
            "type": "object",
            "required": [
              "type"
            ],
            "properties": {
              "type": {
                "enum": [
                  "contains",
                  "not_contains",
                  "matches",
                  "valid_json",
                  "max_chars",
                  "equals_expected"
                ],
                "type": "string"
              },
              "value": {
                "type": "string",
                "description": "Text for contains/not_contains, a regex for matches, a number for max_chars."
              },
              "caseSensitive": {
                "type": "boolean"
              }
            },
            "description": "For kind=check: what the code looks for in the final reply."
          },
          "scaleMax": {
            "type": "integer",
            "maximum": 9007199254740991,
            "minimum": -9007199254740991,
            "description": "Default 5."
          },
          "scaleMin": {
            "type": "integer",
            "maximum": 9007199254740991,
            "minimum": -9007199254740991,
            "description": "Default 1. Ignored for pass_fail and check (always 0..1)."
          },
          "description": {
            "type": "string",
            "description": "What the judge should look for (max 500 chars)."
          }
        }
      },
      "maxItems": 10,
      "minItems": 1,
      "description": "Replacement criteria (1-10) — the whole list, not a diff."
    },
    "rubricId": {
      "type": "string",
      "description": "Rubric id, from caliper_rubrics_list."
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    },
    "visibility": {
      "enum": [
        "PRIVATE",
        "WORKSPACE",
        "SHARED"
      ],
      "type": "string",
      "description": "Who can see it: PRIVATE (only the user), WORKSPACE (every member, the default), or SHARED (specific people, granted afterwards). Say 'make it private' → PRIVATE."
    },
    "description": {
      "anyOf": [
        {
          "type": "string",
          "maxLength": 2000
        },
        {
          "type": "null"
        }
      ],
      "description": "New description; null clears it."
    }
  }
}

Input schema (JSON)

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "sourceId",
    "purpose"
  ],
  "properties": {
    "day": {
      "type": "string",
      "pattern": "^\\d{4}-\\d{2}-\\d{2}$",
      "description": "One UTC day, YYYY-MM-DD."
    },
    "tool": {
      "type": "string",
      "description": "Only traces that called this tool."
    },
    "limit": {
      "type": "integer",
      "maximum": 500,
      "minimum": 1,
      "description": "Most recent N, default 100."
    },
    "purpose": {
      "enum": [
        "review",
        "eval",
        "spec"
      ],
      "type": "string",
      "description": "review | eval | spec."
    },
    "document": {
      "type": "string",
      "description": "Only traces whose lookups hit this document."
    },
    "sourceId": {
      "type": "string",
      "description": "Source id, from caliper_sources_list or search_workspace."
    },
    "datasetId": {
      "type": "string",
      "description": "Add to this dataset…"
    },
    "workspace": {
      "type": "string",
      "description": "Workspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys."
    },
    "approvalId": {
      "type": "string",
      "description": "Approval id from a prior needs_confirmation response. Omit on the first call."
    },
    "errorsOnly": {
      "type": "boolean",
      "description": "Only failed traces."
    },
    "keepAdding": {
      "anyOf": [
        {
          "type": "number",
          "const": 1
        },
        {
          "type": "number",
          "const": 10
        },
        {
          "type": "number",
          "const": 100
        }
      ],
      "description": "Keep adding new matches as they arrive, taking one in this many (1 = every one)."
    },
    "newDatasetName": {
      "type": "string",
      "maxLength": 120,
      "minLength": 2,
      "description": "…or start one with this name."
    }
  }
}