io.github.cyanheads/evals-mcp-server

Return the required and optional fields plus the grader options for a given task_type. Call this before drafting to learn what a record of that type needs. Static — derived from the schemas, with no disk or runtime state.

Read a draft or submitted eval record by id. The id is stable across submit, so it resolves a record whether it is still a draft or already submitted.

Finalize a draft through the committability gate. The server runs deterministic graders against the gold (must PASS), rejects ≥1 declared negative case, requires a recorded, decorrelated independent verification that agrees with the gold, embeds any resolved captures, and checks for duplicates; on pass it flips the record to submitted, stamps submitted_at and a checksum, and freezes it. It refuses with a typed error otherwise and the record stays a draft. For free_response, llm_rubric is not executable by this server; the record is admitted on recorded independent verification and flagged server_verified=false.

Run a deterministic grader spec against one or more candidate answers and get a PASS/REJECT verdict per candidate, plus the resolved comparison value (e.g. the math.js-evaluated numeric target). Use it mid-loop to re-derive or spot-check a gold independently of any saved record. For grader kinds that grade against a reference rather than embedding one (exact_match), supply gold; it is ignored for target-embedding kinds like numeric and mcq. llm_rubric cannot run on this server and requires recorded independent verification before submit.

Delete a draft record by id to clean up an abandoned draft. Draft-only — submitted records are frozen and cannot be discarded. Treat a not_found result on discard as effectively idempotent: a draft that is already gone reports not_found rather than a distinct already-discarded error.

Browse and filter eval records by status, domain, task_type, or tag. Returns a compact summary per record (id, status, task_type, domain, tags, timestamps), not the full records — call evals_get_record for a single record in full. Results are newest-first and the response discloses when the limit truncated the set.

Apply a surgical, field-level patch to a draft using explicit set / append / unset operations by dotted path (e.g. set "grader.rel_tol", append "verification.evidence"), not a full rewrite, so the change stays legible. Returns the updated record, an itemized list of what changed, and a re-run of the self-consistency check since the grader may have moved. Draft-only — submitted records are frozen.

Create a draft eval record carrying its own executable grader. Persists it as a draft, then returns the parsed record parroted back behind a divider, a per-field review protocol, a ready-to-paste verification subagent prompt, and the self-consistency result. It stays draft — self-consistency passing proves the grader discriminates, not that the gold is right; record an independent verification (a subagent report or your own decorrelated check) before evals_submit_draft.

Compile submitted records to a downstream eval format and write the artifact under exports/. Choose jsonl (lossless, the lingua franca), csv (a flattened, lossy spreadsheet summary), inspect (UK AISI Inspect AI), or lm-eval (EleutherAI lm-evaluation-harness). Optionally filter by domain, task_type, or tag. Only submitted records are exported — drafts are skipped. Returns the file path, record count, byte size, and a short preview rather than dumping the full artifact inline.