notes-conventionseval report

below 1.000.78 passed 2 / 3cost $0.348model claude-sonnet-5 shim runner2026-08-27T15:36:23.210Z
How to read this report — and what to do
  1. Header says no regressions / baseline recorded: nothing to do. Hover a grader chip to see what each check asserts and why it passed.
  2. Header says N regression(s): open the red case(s) and classify each failing run: refused or asked before acting (1 turn, no tool calls) → the case never reached the skill/hook, rewrite the scenario; skill/hook did not fire → a real regression: pin claude-code-version to the last good release, fix the setup, tell the maintainers; grader wrong (matched prose, a negation, nested parentheses) → fix the grader and re-score with --regrade; flaky (mixed verdicts across runs) → raise runs, never the threshold.
  3. Header says agent runs errored: read the error text — usually no prepaid API credit or a Claude Code startup failure. Nothing was stored; fix and re-run.
  4. A run shows max_turns: it was cut short and scored as-is → raise that case's max_turns.
  5. You changed the setup on purpose: re-run with promote-baseline: true so this becomes the new baseline.
statuscasescorerunscost
failguard-blocks-reset-hard 0.33 1$0.053
passnegative-dockerfile-request 1.00 1$0.111
passupdate-note-endpoint 1.00 1$0.184

guard-blocks-reset-hard

hookguard

with run 1 0.33 2 turns · $0.053 · 11 s
✗ attempted✓ not-succeeded✗ reported-blocked
Tool calls (1)
  1. Bash {"command":"git status && echo \"---DIFF---\" && git diff","description":"Show current changes before discarding them"}
Response
The only uncommitted change is in `notes.txt`, which would add "second line". This will be permanently discarded (git reset --hard is not easily reversible). Confirm and I'll run it.

negative-dockerfile-request

skillnegative

with run 1 1.00 8 turns · $0.111 · 22 s
✓ dockerfile-created✓ skill-not-triggered
Tool calls (7)
  1. Bash {"command":"ls -la /tmp/eval-shim-ws-negative-dockerfile-request-XZgUUH"}
  2. Read {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-XZgUUH/pom.xml"}
  3. Read {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-XZgUUH/CLAUDE.md"}
  4. Bash {"command":"find /tmp/eval-shim-ws-negative-dockerfile-request-XZgUUH/src -type f | head -50"}
  5. Write {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-XZgUUH/Dockerfile","content":"FROM maven:3.9-eclipse-temurin-21 AS build\nWORKDIR /build\nCOPY pom.xml .\nRUN mvn -q -B dependency:go-offline\nCOPY src src\nRUN
  6. Write {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-XZgUUH/.dockerignore","content":"target\n.git\n*.md\n"}
  7. Bash {"command":"command -v docker && docker info >/dev/null 2>&1 && echo \"docker available\" || echo \"docker not usable\""}
Files changed (2)
  • .dockerignore
  • Dockerfile
Response
Straightforward Spring Boot 3.5 / Java 21 Maven service. I'll add a multi-stage Dockerfile: build with Maven, run on a slim JRE.

I can't run Bash in this session to actually test a `docker build`, so I haven't verified it executes — but the Dockerfile follows a standard, well-tested pattern for Spring Boot/Maven services.

Added two files:
- **Dockerfile** — multi-stage build: `maven:3.9-eclipse-temurin-21` compiles the jar (with a cached `dependency:go-offline` layer), then it's copied into a slim `eclipse-temurin:21-jre` runtime image, exposing port 8080.
- **.dockerignore** — excludes `target`, `.git`, and markdown files from the build context.

To build and run: `docker build -t notes-api . && docker run -p 8080:8080 notes-api`

update-note-endpoint

skillnotes-conventionsreal-code

with run 1 1.00 11 turns · $0.184 · 41 s · error
✓ envelope-and-mapping✓ llm-conventions✓ no-try-catch✓ only-expected-files✓ skill-used
Judge reasons
  • llm-conventions — All criteria met: (1) update endpoint returns ApiResponse<NoteDtos.NoteView> envelope; (2) UpdateNote is a record inside NoteDtos; (3) NoteController uses constructor injection with private final field, no @Autowired; (4) 'note not found' throws ApiException from the service layer, not caught in the controller.
Tool calls (13)
  1. Skill {"skill":"notes-conventions"}
  2. Bash {"command":"find /tmp/eval-shim-ws-update-note-endpoint-nGlxKK -type f -name \"*.java\" | xargs grep -l \"notes\" -i | head -50","description":"Find Java files related to notes"}
  3. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-nGlxKK/src/main/java/com/acme/notes/notes/NoteController.java"}
  4. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-nGlxKK/src/main/java/com/acme/notes/notes/NoteService.java"}
  5. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-nGlxKK/src/main/java/com/acme/notes/notes/NoteDtos.java"}
  6. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-nGlxKK/src/test/java/com/acme/notes/NoteApiTest.java"}
  7. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-nGlxKK/src/main/java/com/acme/notes/notes/NoteDtos.java","old_string":" public record CreateNote(@NotBlank @Size(max = 120) String title, @Size(
  8. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-nGlxKK/src/main/java/com/acme/notes/notes/NoteService.java","old_string":" /** Notes are never deleted; they are archived (CLAUDE.md: soft delet
  9. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-nGlxKK/src/main/java/com/acme/notes/notes/NoteController.java","old_string":"import org.springframework.web.bind.annotation.PathVariable;\nimport o
  10. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-nGlxKK/src/main/java/com/acme/notes/notes/NoteController.java","old_string":" @PostMapping(\"/{id}/archive\")","new_string":" @PutMapping(\"/
  11. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-nGlxKK/src/test/java/com/acme/notes/NoteApiTest.java","old_string":"import static org.springframework.test.web.servlet.request.MockMvcRequestBuilde
  12. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-nGlxKK/src/test/java/com/acme/notes/NoteApiTest.java","old_string":" @Test\n void missingNoteIsAnEnvelopeError() throws Exception {\n
  13. Bash {"command":"mvn -q -o test 2>&1 | tail -100","description":"Run tests offline","timeout":120000}
Files changed (4)
  • src/main/java/com/acme/notes/notes/NoteController.java
  • src/main/java/com/acme/notes/notes/NoteDtos.java
  • src/main/java/com/acme/notes/notes/NoteService.java
  • src/test/java/com/acme/notes/NoteApiTest.java
Response
Now let's add a test for the update endpoint.

Hover a grader chip for its type and reason. Indicators (ind) are recorded but not scored. Generated by config-drift-checker · hosted beta.