6 of 6 cases hold against the baseline. Nothing to do.
overall
1.00from 0.99
cases
6 / 6 at 1.00 · 6 of 6 in suite
model
?moved from claude-sonnet-5
claude code
2.1.296moved from 2.1.258
track
canary
runner
official
cost
$0.92
when
2026-10-12 15:18
What this report checks · 72 grader verdicts across 18 agent runs on 6 cases. A plain claude plugin eval run stops there; the chips are this tool's layer on top:
✓ baseline diff✓ noise bands✓ escalation guards· refusal screening· discovered vs invoked✓ efficiency drift✓ setup health✓ suite completeness
Setup health on Claude Code 2.1.296 · skill frontmatter and eval suite format, checked before the run; these checks start no model runs.
Eval suite format✓ cleanloaded 6 of 6 cases · confirmed by the runner
✓ No setup problems: 1 skill and 6 eval cases pass every static check, and the runner loaded the suite.
How to read this report, and what to do
No drift / baseline recorded: nothing to do. Hover a grader chip to see what each check asserts and why it passed.
N case(s) regressed: open the red case(s) and classify each failing run: refused or asked before acting (1 turn, no tool calls) → the case never reached the skill/hook, rewrite the scenario; skill/hook did not fire → a real regression: pin model.pinned/harness.pinned in .cdc.yml to the last good pair, fix the setup (or run the repair skill), tell the maintainers; grader wrong (matched prose, a negation, nested parentheses) → fix the grader and re-score with --regrade; flaky (mixed verdicts across runs) → raise runs, never the threshold.
Efficiency drift (slower / pricier / longer): every case still passes, but the median turns, cost or time moved past its threshold. Warning by default; add it to fail_on in .cdc.yml to make it red.
Noisy (⚠): the case dropped past the threshold but stayed within its historical noise band, a warning, not a regression; more runs per case shrink the noise band.
Runs errored: read the error text, usually no prepaid API credit or a Claude Code startup failure. Nothing was stored; fix and re-run.
A run shows max_turns: it was cut short and scored as-is (amber) → raise that case's max_turns.
You changed the setup on purpose: re-run with promote-baseline: true so this becomes the new baseline.
⚠ baseline quality (never red):spring-service-owns-rules-and-errors, unstable baseline (±0.20). More runs per case fix this; never loosen the threshold.
Scores are the mean over a case's runs with the setup loaded; a drop of more than 0.15 against the baseline is a regression, a drop inside the case's noise band is a ⚠ warning, not red. Indicators (ind) are recorded but not scored. Generated by config-drift-checker.