config-drift-checker tests your CLAUDE.md, skills and hooks every time Claude Code ships a release, and tells you as soon as one of them stops working.
Open source. It runs in your own GitHub Actions with your own API key, and nothing leaves your repository.
Get it on GitHub Open the demo dashboard| case | before | after | failing graders | |
|---|---|---|---|---|
| guard-blocks-reset-hard | 1.00 | 0.33 | reported-blocked ×3, not-succeeded ×2 | |
| update-note-endpoint | 1.00 | 1.00 | ||
| negative-dockerfile-request | 1.00 | 1.00 |
claude plugin marketplace add jameskomo/config-drift-checker
claude plugin install config-drift-checker@jameskomo
claude
> /config-drift-checker:setup
Your CLAUDE.md, your skills and your hooks. From those it writes the first test cases: a real change to your code, a request that must not trigger a skill, and a command the guard hook must block.
Each case runs several times in a throwaway copy of your repository, with your plugin loaded. Every grader records a verdict and a reason, and a judge model explains what it saw.
A small job checks npm every six hours. When Claude Code ships, the suite runs against the new version, compares the result with your baseline, and turns the check red if something has moved. It can post to Slack as well.
If you already have cases, or only want the check in an existing pipeline, it is one step: uses: jameskomo/config-drift-checker/action@v0. The cases use Anthropic's own claude plugin eval format, so there is nothing of ours to learn and nothing to migrate away from later.
The interesting number is in the second column. The same cases were run with the team's setup loaded and with it removed; the difference is what each part of the setup is actually worth.
| case | with the setup | without | What it means |
|---|---|---|---|
| The guard hook must block git reset --hard | 1.00 | 0.33 | Without the hook, the command usually runs. The hook is doing real work. |
| Add an endpoint across service, DTO and controller | 1.00 | 0.50 | The conventions skill changes how the code comes out. Keep it. |
| A Dockerfile request must not wake the backend skill | 1.00 | 1.00 | No over-triggering. Nothing to fix. |
Three runs per case. The report shows each run, each grader and the judge's reason, so a score is never the end of the story. See the report.
One thing we learned early: a hook test once passed because the model refused the command on its own, before the hook had run. Same score, different cause. That is why the report keeps the reasons.
The plugin and the Action stay free and open. The hosted service is for teams that run this across many repositories and want to see all of it in one place: history across every repository and every Claude Code version, a chart of scores over releases, release-watch without GitHub Actions, and the same suite run against Codex and Cursor.
Agent runs always bill to your own API key. You pay us for the loop around them, never for the model.
| Starter$49 / month | Team$149 / month | Business$399 / month | |
|---|---|---|---|
| Repos | 1 | 10 | unlimited |
| History | 90 days | 1 year | unlimited |
| Agents | Claude Code | Claude Code, Codex, Cursor | Claude Code, Codex, Cursor |
| Release-watch | hosted, 6-hourly | hosted, 6-hourly | hosted, under an hour |
| Reports | per run | per run, plus ablation: what each skill is worth | per run, ablation, audit export for hook enforcement |
| Access | GitHub login | GitHub org | SAML, private runner |
Ask for the beta The first teams get their suite written with them, and the beta at the Starter price.