config-drift-checker · komo-stack

●Baseline recorded · all cases pass

This run is the reference. From now on every run is diffed against it; a drop of more than the threshold turns the check red.

overall: 1.00
cases: 6 / 6 at 1.00
model: ?
claude code: 2.1.296
track: —
runner: official
cost: $0.33
when: 2026-10-28 08:42

What this report checks · 24 grader verdicts across 6 agent runs on 6 cases. A plain claude plugin eval run stops there; the chips are this tool's layer on top:

· baseline diff· noise bands· escalation guards· refusal screening· discovered vs invoked· efficiency drift· suite completeness

How to read this report, and what to do

No drift / baseline recorded: nothing to do. Hover a grader chip to see what each check asserts and why it passed.
N case(s) regressed: open the red case(s) and classify each failing run: refused or asked before acting (1 turn, no tool calls) → the case never reached the skill/hook, rewrite the scenario; skill/hook did not fire → a real regression: pin model.pinned/harness.pinned in .cdc.yml to the last good pair, fix the setup (or run the repair skill), tell the maintainers; grader wrong (matched prose, a negation, nested parentheses) → fix the grader and re-score with --regrade; flaky (mixed verdicts across runs) → raise runs, never the threshold.
Efficiency drift (slower / pricier / longer): every case still passes, but the median turns, cost or time moved past its threshold. Warning by default; add it to fail_on in .cdc.yml to make it red.
Noisy (⚠): the case dropped past the threshold but stayed within its historical noise band, a warning, not a regression; more runs per case shrink the noise band.
Runs errored: read the error text, usually no prepaid API credit or a Claude Code startup failure. Nothing was stored; fix and re-run.
A run shows max_turns: it was cut short and scored as-is (amber) → raise that case's max_turns.
You changed the setup on purpose: re-run with promote-baseline: true so this becomes the new baseline.

status	case	score	95% range	turns	cost	runs
pass	guard-blocks-destructive-git	1.00	needs 3+ runs	2	$0.06	1
pass	spring-controller-follows-conventions	1.00	needs 3+ runs	3	$0.05	1
pass	spring-request-dtos-are-validated-records	1.00	needs 3+ runs	3	$0.06	1
pass	spring-service-owns-rules-and-errors	1.00	needs 3+ runs	3	$0.06	1
pass	spring-work-triggers-skill	1.00	needs 3+ runs	3	$0.05	1
pass	vue-request-does-not-trigger-skill	1.00	needs 3+ runs	1	$0.04	1

95% range (hover the column): where the true value most likely sits, given how much the runs varied (right 95 times in 100). More runs give a narrower range. Information only: no verdict here uses it.

show failing and flagged runs only

guard-blocks-destructive-git

pass · 1.00 · passed 1 of 1 (0.21 to 1.00)

What this case evaluates

The checks (4)

`attempted`	tool_used	tool used 1+ times
`blocked-by-hook`	regex	must contain in last_message
`no-reset-succeeded`	regex	must contain in last_message
`reported-blocked`	regex	must contain in last_message

withrun 11.00 2 turns · $0.06 · 5.0 s

· attempted· blocked-by-hook· no-reset-succeeded· reported-blocked

Tool calls (0)

none

Response

spring-controller-follows-conventions

pass · 1.00 · passed 1 of 1 (0.21 to 1.00)

What this case evaluates

The checks (6)

`api-response-wrapper`	regex	must contain in last_message
`constructor-injection`	regex	must contain in last_message
`no-field-injection`	regex	must contain in last_message
`page-clamping`	regex	must contain in last_message
`reviewer-judgment`	llm	judge model:
`skill-fired`	tool_used	tool used 1+ times

withrun 11.00 3 turns · $0.05 · 15 s

· api-response-wrapper· constructor-injection· no-field-injection· page-clamping· reviewer-judgment· skill-fired

Tool calls (0)

none

Response

spring-request-dtos-are-validated-records

pass · 1.00 · passed 1 of 1 (0.21 to 1.00)

What this case evaluates

The checks (6)

`constraint-annotations`	regex	must contain in last_message
`dtos-holder`	regex	must contain in last_message
`record-shapes`	regex	must contain in last_message
`reviewer-judgment`	llm	judge model:
`skill-fired`	tool_used	tool used 1+ times
`valid-annotation`	regex	must contain in last_message

withrun 11.00 3 turns · $0.06 · 18 s

· constraint-annotations· dtos-holder· record-shapes· reviewer-judgment· skill-fired· valid-annotation

Tool calls (0)

none

Response

spring-service-owns-rules-and-errors

pass · 1.00 · passed 1 of 1 (0.21 to 1.00)

What this case evaluates

The checks (5)

`advice-handler`	regex	must contain in last_message
`no-map-result`	regex	must contain in last_message
`reviewer-judgment`	llm	judge model:
`skill-fired`	tool_used	tool used 1+ times
`throws-domain-exception`	regex	must contain in last_message

withrun 11.00 3 turns · $0.06 · 22 s

· advice-handler· no-map-result· reviewer-judgment· skill-fired· throws-domain-exception

Tool calls (0)

none

Response

spring-work-triggers-skill

pass · 1.00 · passed 1 of 1 (0.21 to 1.00)

What this case evaluates

The checks (1)

skill-fired

tool_used

tool used 1+ times

withrun 11.00 3 turns · $0.05 · 11 s

· skill-fired

Tool calls (0)

none

Response

vue-request-does-not-trigger-skill

pass · 1.00 · passed 1 of 1 (0.21 to 1.00)

What this case evaluates

The checks (2)

`is-pinia-store`	regex	must contain in last_message
`skill-not-fired`	tool_used	tool used 1+ times

withrun 11.00 1 turns · $0.04 · 4.0 s

· is-pinia-store· skill-not-fired

Tool calls (0)

none

Response

Scores are the mean over a case's runs with the setup loaded; a drop of more than 0.15 against the baseline is a regression. Indicators (ind) are recorded but not scored. Generated by config-drift-checker.