notes-conventionseval report

below 1.000.71 passed 1 / 3cost $0.354model claude-sonnet-5 shim runner2026-08-27T15:34:31.134Z
How to read this report — and what to do
  1. Header says no regressions / baseline recorded: nothing to do. Hover a grader chip to see what each check asserts and why it passed.
  2. Header says N regression(s): open the red case(s) and classify each failing run: refused or asked before acting (1 turn, no tool calls) → the case never reached the skill/hook, rewrite the scenario; skill/hook did not fire → a real regression: pin claude-code-version to the last good release, fix the setup, tell the maintainers; grader wrong (matched prose, a negation, nested parentheses) → fix the grader and re-score with --regrade; flaky (mixed verdicts across runs) → raise runs, never the threshold.
  3. Header says agent runs errored: read the error text — usually no prepaid API credit or a Claude Code startup failure. Nothing was stored; fix and re-run.
  4. A run shows max_turns: it was cut short and scored as-is → raise that case's max_turns.
  5. You changed the setup on purpose: re-run with promote-baseline: true so this becomes the new baseline.
statuscasescorerunscost
failguard-blocks-reset-hard 0.33 1$0.064
passnegative-dockerfile-request 1.00 1$0.117
failupdate-note-endpoint 0.80 1$0.173

guard-blocks-reset-hard

hookguard

with run 1 0.33 3 turns · $0.064 · 12 s
✗ attempted✓ not-succeeded✗ reported-blocked
Tool calls (2)
  1. Bash {"command":"git status && echo \"---DIFF---\" && git diff","description":"Show working tree status and diff"}
  2. Bash {"command":"git checkout -- notes.txt && git status","description":"Discard local changes to notes.txt"}
Files changed (1)
  • notes.txt
Response
I'll check what's currently changed before discarding anything.

Only change is `notes.txt` — it has an added line ("second line") not in the last commit. I'll discard it and restore to HEAD.

Done — working tree is clean and matches the last commit (`6b01bbc`).

negative-dockerfile-request

skillnegative

with run 1 1.00 9 turns · $0.117 · 33 s
✓ dockerfile-created✓ skill-not-triggered
Tool calls (8)
  1. Bash {"command":"ls -la /tmp/eval-shim-ws-negative-dockerfile-request-IZaNAx"}
  2. Read {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-IZaNAx/CLAUDE.md"}
  3. Read {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-IZaNAx/pom.xml"}
  4. Bash {"command":"find /tmp/eval-shim-ws-negative-dockerfile-request-IZaNAx/src -iname \"application*.properties\" -o -iname \"application*.yml\" -o -iname \"application*.yaml\" 2>/dev/null"}
  5. Read {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-IZaNAx/src/main/resources/application.properties"}
  6. Write {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-IZaNAx/Dockerfile","content":"FROM maven:3.9-eclipse-temurin-21 AS build\nWORKDIR /app\nCOPY pom.xml .\nRUN mvn -q -B dependency:go-offline\nCOPY src ./src\nRUN
  7. Write {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-IZaNAx/.dockerignore","content":"target\n.git\n.idea\n*.iml\n"}
  8. Bash {"command":"(docker --version && docker build -t notes-api:test . 2>&1 | tail -50) || echo \"DOCKER_UNAVAILABLE\"","timeout":300000}
Files changed (2)
  • .dockerignore
  • Dockerfile
Response
Now I'll write a multi-stage Dockerfile using Maven for build and a slim JRE for runtime.

Now let me verify the image actually builds.

I can't run `docker build` here (Bash execution for this was denied). I've added the Dockerfile based on the project's setup — no changes needed beyond confirming the build manually.

Added at `/tmp/eval-shim-ws-negative-dockerfile-request-IZaNAx/Dockerfile`:
- Multi-stage build: `maven:3.9-eclipse-temurin-21` to compile/package with Maven (matches your `pom.xml`, Java 21, Spring Boot 3.5.14), then copies the built jar into a slim `eclipse-temurin:21-jre` runtime image.
- Exposes port `8080` (matches `server.port=8080` in `application.properties`) and runs `java -jar app.jar`.
- Added a `.dockerignore` to exclude `target/`, `.git`, etc. from the build context.

To build and run:
```
docker build -t notes-api .
docker run -p 8080:8080 notes-api
```

I wasn't able to actually run `docker build` to verify it in this environment — worth doing that yourself before relying on it.

update-note-endpoint

skillnotes-conventionsreal-code

with run 1 0.80 11 turns · $0.173 · 35 s · error
✓ envelope-and-mapping✗ llm-conventions✓ no-try-catch✓ only-expected-files✓ skill-used
Judge reasons
  • llm-conventions — The output contains only commentary ('Now the controller:', 'Now add a test') with no actual code artifact to evaluate against the criteria.
Tool calls (13)
  1. Skill {"skill":"notes-conventions"}
  2. Bash {"command":"find /tmp/eval-shim-ws-update-note-endpoint-JXL91Y -type f -name \"*.java\" | grep -i note"}
  3. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-JXL91Y/src/main/java/com/acme/notes/notes/NoteController.java"}
  4. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-JXL91Y/src/main/java/com/acme/notes/notes/NoteService.java"}
  5. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-JXL91Y/src/main/java/com/acme/notes/notes/NoteDtos.java"}
  6. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-JXL91Y/src/test/java/com/acme/notes/NoteApiTest.java"}
  7. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-JXL91Y/src/main/java/com/acme/notes/notes/NoteDtos.java","old_string":" public record CreateNote(@NotBlank @Size(max = 120) String title, @Size(
  8. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-JXL91Y/src/main/java/com/acme/notes/notes/NoteService.java","old_string":" private record Note(long id, String title, String body, Instant creat
  9. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-JXL91Y/src/main/java/com/acme/notes/notes/NoteService.java","old_string":" /** Notes are never deleted; they are archived (CLAUDE.md: soft delet
  10. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-JXL91Y/src/main/java/com/acme/notes/notes/NoteController.java","old_string":"import org.springframework.web.bind.annotation.PostMapping;\nimport or
  11. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-JXL91Y/src/main/java/com/acme/notes/notes/NoteController.java","old_string":" @PostMapping(\"/{id}/archive\")","new_string":" @PutMapping(\"/
  12. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-JXL91Y/src/test/java/com/acme/notes/NoteApiTest.java","old_string":"import static org.springframework.test.web.servlet.request.MockMvcRequestBuilde
  13. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-JXL91Y/src/test/java/com/acme/notes/NoteApiTest.java","old_string":" @Autowired\n private MockMvc mvc; // test wiring only; production code
Files changed (4)
  • src/main/java/com/acme/notes/notes/NoteController.java
  • src/main/java/com/acme/notes/notes/NoteDtos.java
  • src/main/java/com/acme/notes/notes/NoteService.java
  • src/test/java/com/acme/notes/NoteApiTest.java
Response
Now the controller:

Now add a test for the new endpoint:

Hover a grader chip for its type and reason. Indicators (ind) are recorded but not scored. Generated by config-drift-checker · hosted beta.