Benchmark your own prompt on 100+ LLMs with real API calls; pick the cheapest model that passes.
Maintain this server? Claim it
Call this server's tools from your browser. Nothing to install.
Bearer followed by your personal key (om_...) from openmark.ai/ui, account menu > API keys
Requests go through mcptry because browsers can't call most MCP servers directly. Header values are sent to the server for that request only and are never stored or logged.
Tool list from our last check. Connect live to refresh it.
Credits a run_benchmark call would use, before spending anything: the expected total, the split per model, the balance needed to start, the worst case, and the user's balance. Same inputs and run settings as run_benchmark. Use it to drop an expensive model, or to compare settings (e.g. max_tokens), before running.
The full task YAML (a JSON object with the same fields also works)
Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it
Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures
Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value
From prepare_attachments, so image and document tokens are counted
Seconds per answer, with timeout_profile custom
Passes per model; 2+ shows which models are consistent
With temperature_discovery: which temperature wins a tie
How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout
Find each model's best temperature for this task. Costs about 6x as much
This tool isn't marked read-only, so it may change something on the server or in an account you connected.
Credits a run_benchmark call would use, before spending anything: the expected total, the split per model, the balance needed to start, the worst case, and the user's balance. Same inputs and run settings as run_benchmark. Use it to drop an expensive model, or to compare settings (e.g. max_tokens), before running.
| Argument | Type | Description |
|---|---|---|
| models* | array | |
| preset | string | Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it |
| fail_fast | boolean | Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures |
| task_yaml* | string | The full task YAML (a JSON object with the same fields also works) |
| max_tokens | integer | Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value |
| upload_uid | string | From prepare_attachments, so image and document tokens are counted |
| custom_timeout | integer | Seconds per answer, with timeout_profile custom |
| stability_runs | integer | Passes per model; 2+ shows which models are consistent |
| temp_preference | string | With temperature_discovery: which temperature wins a tie |
| timeout_profile | string | How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout |
| temperature_discovery | boolean | Find each model's best temperature for this task. Costs about 6x as much |
{
"type": "object",
"required": [
"task_yaml",
"models"
],
"properties": {
"models": {
"type": "array",
"items": {
"type": "string"
},
"maxItems": 20,
"minItems": 1
},
"preset": {
"enum": [
"standard",
"thorough"
],
"type": "string",
"description": "Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it"
},
"fail_fast": {
"type": "boolean",
"default": true,
"description": "Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures"
},
"task_yaml": {
"type": "string",
"description": "The full task YAML (a JSON object with the same fields also works)"
},
"max_tokens": {
"type": "integer",
"default": 500,
"maximum": 16000,
"minimum": 0,
"description": "Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value"
},
"upload_uid": {
"type": "string",
"description": "From prepare_attachments, so image and document tokens are counted"
},
"custom_timeout": {
"type": "integer",
"maximum": 600,
"minimum": 120,
"description": "Seconds per answer, with timeout_profile custom"
},
"stability_runs": {
"type": "integer",
"default": 2,
"maximum": 5,
"minimum": 1,
"description": "Passes per model; 2+ shows which models are consistent"
},
"temp_preference": {
"enum": [
"low",
"high"
],
"type": "string",
"default": "low",
"description": "With temperature_discovery: which temperature wins a tie"
},
"timeout_profile": {
"enum": [
"snappy",
"patient",
"custom"
],
"type": "string",
"default": "snappy",
"description": "How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout"
},
"temperature_discovery": {
"type": "boolean",
"default": false,
"description": "Find each model's best temperature for this task. Costs about 6x as much"
}
}
}The user's plan, credit balance and limits, and what a credit pack, Pro or Expert would unlock. Use it before large runs, or when a run is refused for credits or limits.
No arguments.
{
"type": "object",
"properties": {}
}Status of a benchmark job, and once completed the ranked results (score, stability, cost per run, latency), the tests each model missed with its answer, a recommendation for the chosen objective and quick picks for the others.
| Argument | Type | Description |
|---|---|---|
| job_id* | string | |
| tolerance | number | Score points below the top score that still count as good enough |
| show_misses | boolean | List the tests each model got wrong, with the expected answer and the model's reply |
| optimize_for | string | accuracy (same order as the app's results table): best score, then cheapest, then fastest. cost / speed / stability: cheapest / fastest / most consistent model within `tolerance` points of the top score |
{
"type": "object",
"required": [
"job_id"
],
"properties": {
"job_id": {
"type": "string"
},
"tolerance": {
"type": "number",
"default": 2,
"maximum": 50,
"minimum": 0,
"description": "Score points below the top score that still count as good enough"
},
"show_misses": {
"type": "boolean",
"default": true,
"description": "List the tests each model got wrong, with the expected answer and the model's reply"
},
"optimize_for": {
"enum": [
"accuracy",
"cost",
"speed",
"stability"
],
"type": "string",
"default": "accuracy",
"description": "accuracy (same order as the app's results table): best score, then cheapest, then fastest. cost / speed / stability: cheapest / fastest / most consistent model within `tolerance` points of the top score"
}
}
}How to write an OpenMark task: the YAML structure, all 18 scoring modes, and tips. Call this before writing a task. Pass a topic for the full detail on one subject (the app's Task Guide): fields, scoring, composite, chaining, documents, examples, best_practices, or a scoring mode name.
| Argument | Type | Description |
|---|---|---|
| topic | string | Optional: fields, scoring, composite, chaining, documents, examples, best_practices, or a scoring mode name such as regex_match |
{
"type": "object",
"properties": {
"topic": {
"type": "string",
"description": "Optional: fields, scoring, composite, chaining, documents, examples, best_practices, or a scoring mode name such as regex_match"
}
}
}Current OpenMark model ids with list prices ($ per 1M tokens), as text and as a structured list. Use these ids in run_benchmark. Append @low, @medium or @high to a reasoning model id to set its reasoning effort.
| Argument | Type | Description |
|---|---|---|
| search | string | Filter by id, name or provider, e.g. 'claude', 'gemini flash' |
| images_only | boolean | Only models that accept image attachments |
| max_output_price | number | Only models at or below this $ per 1M output tokens |
{
"type": "object",
"properties": {
"search": {
"type": "string",
"description": "Filter by id, name or provider, e.g. 'claude', 'gemini flash'"
},
"images_only": {
"type": "boolean",
"description": "Only models that accept image attachments"
},
"max_output_price": {
"type": "number",
"description": "Only models at or below this $ per 1M output tokens"
}
}
}Get an upload_uid and a one-hour upload link (no API key needed) for images, PDFs or text documents, with the exact curl command. Upload the files, reference them by file name in each test's attachments list, then pass the upload_uid to estimate_cost / run_benchmark.
| Argument | Type | Description |
|---|---|---|
| file_names | array | Optional: the local file paths to upload, to get a ready-to-run command |
{
"type": "object",
"properties": {
"file_names": {
"type": "array",
"items": {
"type": "string"
},
"maxItems": 20,
"description": "Optional: the local file paths to upload, to get a ready-to-run command"
}
}
}Save a task (YAML, see get_task_format) to the user's OpenMark account and run it on the given models with real API calls. Returns a job_id and the expected cost; poll get_results. Spends the user's credits; pass max_credits to only start within a budget. Optional run settings match the app's Configure & Run panel (preset, stability_runs, max_tokens, temperature_discovery, temp_preference, timeout_profile, custom_timeout, fail_fast).
| Argument | Type | Description |
|---|---|---|
| models* | array | Model ids from list_models |
| preset | string | Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it |
| fail_fast | boolean | Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures |
| task_yaml* | string | The full task YAML (a JSON object with the same fields also works) |
| max_tokens | integer | Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value |
| upload_uid | string | From prepare_attachments, when tests use attachments |
Check a task YAML the way OpenMark will before running it: structure, scoring settings, and the test limit of the user's plan. Free, saves nothing. Returns the problems to fix and warnings worth a look.
| Argument | Type | Description |
|---|---|---|
| task_yaml* | string | The full task YAML (a JSON object with the same fields also works) |
{
"type": "object",
"required": [
"task_yaml"
],
"properties": {
"task_yaml": {
"type": "string",
"description": "The full task YAML (a JSON object with the same fields also works)"
}
}
}Watch the task of a completed job: when new or cheaper models ship, OpenMark re-tests up to 4 of them next to the user's pick and emails the user only when one beats it. Each check uses the user's credits at the normal rate.
| Argument | Type | Description |
|---|---|---|
| model | string | The model to keep as the pick (default: the best-scoring model). Use the model you recommended. |
| job_id* | string |
{
"type": "object",
"required": [
"job_id"
],
"properties": {
"model": {
"type": "string",
"description": "The model to keep as the pick (default: the best-scoring model). Use the model you recommended."
},
"job_id": {
"type": "string"
}
}
}| max_credits |
| integer |
| Budget: if the expected cost is higher, nothing runs and you get the quote instead |
| custom_timeout | integer | Seconds per answer, with timeout_profile custom |
| stability_runs | integer | Passes per model; 2+ shows which models are consistent |
| temp_preference | string | With temperature_discovery: which temperature wins a tie |
| timeout_profile | string | How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout |
| temperature_discovery | boolean | Find each model's best temperature for this task. Costs about 6x as much |
{
"type": "object",
"required": [
"task_yaml",
"models"
],
"properties": {
"models": {
"type": "array",
"items": {
"type": "string"
},
"maxItems": 20,
"minItems": 1,
"description": "Model ids from list_models"
},
"preset": {
"enum": [
"standard",
"thorough"
],
"type": "string",
"description": "Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it"
},
"fail_fast": {
"type": "boolean",
"default": true,
"description": "Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures"
},
"task_yaml": {
"type": "string",
"description": "The full task YAML (a JSON object with the same fields also works)"
},
"max_tokens": {
"type": "integer",
"default": 500,
"maximum": 16000,
"minimum": 0,
"description": "Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value"
},
"upload_uid": {
"type": "string",
"description": "From prepare_attachments, when tests use attachments"
},
"max_credits": {
"type": "integer",
"minimum": 1,
"description": "Budget: if the expected cost is higher, nothing runs and you get the quote instead"
},
"custom_timeout": {
"type": "integer",
"maximum": 600,
"minimum": 120,
"description": "Seconds per answer, with timeout_profile custom"
},
"stability_runs": {
"type": "integer",
"default": 2,
"maximum": 5,
"minimum": 1,
"description": "Passes per model; 2+ shows which models are consistent"
},
"temp_preference": {
"enum": [
"low",
"high"
],
"type": "string",
"default": "low",
"description": "With temperature_discovery: which temperature wins a tie"
},
"timeout_profile": {
"enum": [
"snappy",
"patient",
"custom"
],
"type": "string",
"default": "snappy",
"description": "How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout"
},
"temperature_discovery": {
"type": "boolean",
"default": false,
"description": "Find each model's best temperature for this task. Costs about 6x as much"
}
}
}