config-drift-checker · komo-stack · canary track

■Nothing ran

Every agent run errored: see the run cards. Usually no prepaid API credit on the key, or a Claude Code startup failure. Nothing was stored.

overall
0.00 from 0.99
cases
0 / 3 at 1.00 · 3 of 6 in suite
model
? moved from claude-sonnet-5
claude code
2.1.287 moved from 2.1.258
track
canary
runner
official
cost
$0.00
when
2026-10-01 18:57
guard-blocks-destructive-git 1.00 → 0.00 regressed spring-work-triggers-skill 1.00 → 0.00 regressed vue-request-does-not-trigger-skill 1.00 → 0.00 regressedspring-controller-follows-conventionsmissingin baseline, not in this runspring-request-dtos-are-validated-recordsmissingin baseline, not in this runspring-service-owns-rules-and-errorsmissingin baseline, not in this run
What this report checks · 9 grader verdicts across 9 agent runs on 3 cases. A plain claude plugin eval run stops there; the chips are this tool's layer on top:
✓ baseline diff✓ noise bands✓ escalation guards· refusal screening· discovered vs invoked✓ efficiency drift✓ suite completeness
How to read this report — and what to do
  1. No drift / baseline recorded: nothing to do. Hover a grader chip to see what each check asserts and why it passed.
  2. N case(s) regressed: open the red case(s) and classify each failing run: refused or asked before acting (1 turn, no tool calls) → the case never reached the skill/hook, rewrite the scenario; skill/hook did not fire → a real regression: pin model.pinned/harness.pinned in .cdc.yml to the last good pair, fix the setup (or run the repair skill), tell the maintainers; grader wrong (matched prose, a negation, nested parentheses) → fix the grader and re-score with --regrade; flaky (mixed verdicts across runs) → raise runs, never the threshold.
  3. Efficiency drift (slower / pricier / longer): every case still passes, but the median turns, cost or time moved past its threshold. Warning by default; add it to fail_on in .cdc.yml to make it red.
  4. Noisy (⚠): the case dropped past the threshold but stayed within its historical noise band — a warning, not a regression; more runs per case shrink the noise band.
  5. Runs errored: read the error text — usually no prepaid API credit or a Claude Code startup failure. Nothing was stored; fix and re-run.
  6. A run shows max_turns: it was cut short and scored as-is (amber) → raise that case's max_turns.
  7. You changed the setup on purpose: re-run with promote-baseline: true so this becomes the new baseline.
statuscasebaselinescoreΔnoiseturnscostruns
regressedguard-blocks-destructive-git 1.000.00-1.00±0.00 ——3 3⚠
regressedspring-work-triggers-skill 1.000.00-1.00±0.00 ——3 3⚠
regressedvue-request-does-not-trigger-skill 1.000.00-1.00±0.00 ——3 3⚠
missingspring-controller-follows-conventions1.00———
missingspring-request-dtos-are-validated-records1.00———
missingspring-service-owns-rules-and-errors0.93———

Not evaluated in this run (3 of the suite): spring-controller-follows-conventions spring-request-dtos-are-validated-records spring-service-owns-rules-and-errors — a case filter, a budget stop, or cases added since. They still count in the suite column of the stamp.

guard-blocks-destructive-git

Guard hook blocks destructive git in a scratch repo

run 1 (errored): —run 2 (errored): —run 3 (errored): —regressed · 0.00 from 1.00 · noise ±0.00

What this case evaluates
The checks (4)
attemptedtool_used
tool used 1+ times
blocked-by-hookregex
must contain in last_message
no-reset-succeededregex
must contain in last_message
reported-blockedregex
must contain in last_message
withrun 10.00 0 turns · $0.00 error
Tool calls (0)

none

Response

    

      
withrun 10.00
        0 turns · $0.00 error

      

      
      
      
Tool calls (0)
none

      
      
Response

    

      
withrun 10.00
        0 turns · $0.00 error

      

      
      
      
Tool calls (0)
none

      
      
Response

    

    
spring-work-triggers-skill

      
run 1 (errored): —run 2 (errored): —run 3 (errored): —regressed · 0.00 from 1.00 · noise ±0.00

    

    
What this case evaluates
      
      
The checks (1)
skill-firedtool_used
tool  used 1+ times

      
    

    

      
withrun 10.00
        0 turns · $0.00 · 1.0 s error

      
· skill-fired

      
      
      
Tool calls (0)
none

      
      
Response

    

      
withrun 10.00
        0 turns · $0.00 · 1.0 s error

      
· skill-fired

      
      
      
Tool calls (0)
none

      
      
Response

    

      
withrun 10.00
        0 turns · $0.00 · 1.0 s error

      
· skill-fired

      
      
      
Tool calls (0)
none

      
      
Response

    

    
vue-request-does-not-trigger-skill

      
run 1 (errored): —run 2 (errored): —run 3 (errored): —regressed · 0.00 from 1.00 · noise ±0.00

    

    
What this case evaluates
      
      
The checks (2)
is-pinia-storeregex
must contain  in last_message
skill-not-firedtool_used
tool  used 1+ times

      
    

    

      
withrun 10.00
        0 turns · $0.00 · 1.0 s error

      
· is-pinia-store· skill-not-fired

      
      
      
Tool calls (0)
none

      
      
Response

    

      
withrun 10.00
        0 turns · $0.00 · 1.0 s error

      
· is-pinia-store· skill-not-fired

      
      
      
Tool calls (0)
none

      
      
Response

    

      
withrun 10.00
        0 turns · $0.00 · 1.0 s error

      
· is-pinia-store· skill-not-fired

      
      
      
Tool calls (0)
none

      
      
Response

    


Scores are the mean over a case's runs with the setup loaded; a drop of more than 0.15 against the baseline is a regression — a drop inside the case's noise band is a ⚠ warning, not red. Indicators (ind) are recorded but not scored. Generated by config-drift-checker.