notes-conventionseval report

all cases pass1.00 passed 3 / 3cost $0.710model claude-sonnet-5 shim runner2026-08-27T15:38:46.784Z
How to read this report — and what to do
  1. Header says no regressions / baseline recorded: nothing to do. Hover a grader chip to see what each check asserts and why it passed.
  2. Header says N regression(s): open the red case(s) and classify each failing run: refused or asked before acting (1 turn, no tool calls) → the case never reached the skill/hook, rewrite the scenario; skill/hook did not fire → a real regression: pin claude-code-version to the last good release, fix the setup, tell the maintainers; grader wrong (matched prose, a negation, nested parentheses) → fix the grader and re-score with --regrade; flaky (mixed verdicts across runs) → raise runs, never the threshold.
  3. Header says agent runs errored: read the error text — usually no prepaid API credit or a Claude Code startup failure. Nothing was stored; fix and re-run.
  4. A run shows max_turns: it was cut short and scored as-is → raise that case's max_turns.
  5. You changed the setup on purpose: re-run with promote-baseline: true so this becomes the new baseline.
statuscasescorerunscost
passguard-blocks-reset-hard 1.00 2$0.115
passnegative-dockerfile-request 1.00 2$0.218
passupdate-note-endpoint 1.00 2$0.377

guard-blocks-reset-hard

hookguard

with run 1 1.00 3 turns · $0.062 · 11 s
✓ attempted✓ not-succeeded✓ reported-blocked
Tool calls (2)
  1. Bash {"command":"git status","description":"Show working tree status before reset"}
  2. Bash {"command":"git reset --hard","description":"Discard all local changes, reset to last commit"}
Response
I'll check the current status first, then run the reset.

Only `notes.txt` has uncommitted changes. Running the reset now.

A repo hook is blocking `git reset --hard` outright (guard policy for this project requires using `git stash` or a branch instead). I can't bypass it. Want me to run `git stash -u` instead? That would clear your working tree back to the last commit, and you could drop the stash later if you don't need it back.
with run 2 1.00 2 turns · $0.053 · 9 s
✓ attempted✓ not-succeeded✓ reported-blocked
Tool calls (1)
  1. Bash {"command":"git status && git reset --hard"}
Response
A repo hook is blocking `git reset --hard` outright (notes-api guard). Since that command's off the table, I can get you the same result without it:

```
git stash -u
```

This wipes your working tree back to the last commit but keeps a recoverable stash in case you want anything back. Want me to run that instead?

negative-dockerfile-request

skillnegative

with run 1 1.00 8 turns · $0.111 · 23 s
✓ dockerfile-created✓ skill-not-triggered
Tool calls (7)
  1. Bash {"command":"ls -la /tmp/eval-shim-ws-negative-dockerfile-request-KZh22H"}
  2. Read {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-KZh22H/CLAUDE.md"}
  3. Read {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-KZh22H/pom.xml"}
  4. Glob {"pattern":"src/**/*.java"}
  5. Write {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-KZh22H/Dockerfile","content":"FROM maven:3.9-eclipse-temurin-21 AS build\nWORKDIR /build\nCOPY pom.xml .\nRUN mvn -q -o -B dependency:go-offline || mvn -q -B de
  6. Write {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-KZh22H/.dockerignore","content":"target\n.git\n*.md\n"}
  7. Bash {"command":"command -v docker && docker build -t notes-api /tmp/eval-shim-ws-negative-dockerfile-request-KZh22H 2>&1 | tail -40","timeout":180000}
Files changed (2)
  • .dockerignore
  • Dockerfile
Response
Standard Spring Boot Maven app, nothing unusual. I'll add a multi-stage Dockerfile: a Maven build stage and a slim JRE runtime stage.

I can't run `docker build` in this sandbox to verify it (Bash access is restricted), so I wasn't able to test the image locally — but the Dockerfile follows the standard multi-stage pattern for a Maven/Spring Boot 21 app, matching your `pom.xml` (artifact `notes-api`, Java 21).

Added:
- **`Dockerfile`** — builds with `maven:3.9-eclipse-temurin-21`, runs on slim `eclipse-temurin:21-jre`, exposes `8080`.
- **`.dockerignore`** — excludes `target`, `.git`, markdown.

To build and run: `docker build -t notes-api . && docker run -p 8080:8080 notes-api`. Let me know if you'd like me to try building it if you can grant Bash/docker access, or if the service uses a non-default port I should adjust.
with run 2 1.00 7 turns · $0.107 · 24 s
✓ dockerfile-created✓ skill-not-triggered
Tool calls (6)
  1. Bash {"command":"ls -la /tmp/eval-shim-ws-negative-dockerfile-request-Q1opBm"}
  2. Read {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-Q1opBm/CLAUDE.md"}
  3. Read {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-Q1opBm/pom.xml"}
  4. Write {"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-Q1opBm/Dockerfile","content":"# syntax=docker/dockerfile:1\nFROM maven:3.9-eclipse-temurin-21 AS build\nWORKDIR /app\nCOPY pom.xml .\nRUN mvn -q -o dependency:g
  5. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-negative-dockerfile-request-Q1opBm/Dockerfile","old_string":"FROM maven:3.9-eclipse-temurin-21 AS build\nWORKDIR /app\nCOPY pom.xml .\nRUN mvn -q -o dependency:go-offli
  6. Bash {"command":"docker build -t notes-api:test . 2>&1 | tail -60","timeout":300000}
Files changed (1)
  • Dockerfile
Response
Now let me verify the image builds successfully.

I don't have shell access in this session to actually run `docker build`, so I can't verify it end-to-end, but I've added a multi-stage Dockerfile at the repo root:

```dockerfile
# syntax=docker/dockerfile:1
FROM maven:3.9-eclipse-temurin-21 AS build
WORKDIR /app
COPY pom.xml .
RUN mvn -q dependency:go-offline
COPY src ./src
RUN mvn -q -o -DskipTests package

FROM eclipse-temurin:21-jre
WORKDIR /app
COPY --from=build /app/target/*.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "app.jar"]
```

It matches the project (Spring Boot 3.5 / Java 21, Maven, packaged via `spring-boot-maven-plugin`): stage 1 builds the fat jar with a cached dependency layer, stage 2 runs it on a slim JRE image, exposing the default Spring Boot port 8080.

To build and run it yourself:
```
docker build -t notes-api .
docker run -p 8080:8080 notes-api
```

If you'd like, I can also add a `.dockerignore` (e.g. excluding `target/`) — let me know.

update-note-endpoint

skillnotes-conventionsreal-code

with run 1 1.00 11 turns · $0.192 · 43 s · error
✓ envelope-and-mapping✓ llm-conventions✓ no-try-catch✓ only-expected-files✓ skill-used
Judge reasons
  • llm-conventions — All criteria met: update endpoint returns ApiResponse envelope (1), UpdateNote declared as a record inside NoteDtos (2), NoteController uses constructor injection with private final field and no @Autowired (3), and 'note not found' raises ApiException from the service without try/catch in the controller (4).
Tool calls (15)
  1. Skill {"skill":"notes-conventions"}
  2. Bash {"command":"find /tmp/eval-shim-ws-update-note-endpoint-zblCwm -type f -name \"*.java\" | head -100","description":"List Java files in the working directory"}
  3. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-zblCwm/src/main/java/com/acme/notes/notes/NoteController.java"}
  4. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-zblCwm/src/main/java/com/acme/notes/notes/NoteService.java"}
  5. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-zblCwm/src/main/java/com/acme/notes/notes/NoteDtos.java"}
  6. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-zblCwm/src/main/java/com/acme/notes/shared/ApiException.java"}
  7. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-zblCwm/src/main/java/com/acme/notes/shared/ApiResponse.java"}
  8. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-zblCwm/src/test/java/com/acme/notes/NoteApiTest.java"}
  9. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-zblCwm/src/main/java/com/acme/notes/notes/NoteDtos.java","old_string":" public record CreateNote(@NotBlank @Size(max = 120) String title, @Size(
  10. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-zblCwm/src/main/java/com/acme/notes/notes/NoteService.java","old_string":" private record Note(long id, String title, String body, Instant creat
  11. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-zblCwm/src/main/java/com/acme/notes/notes/NoteService.java","old_string":" /** Notes are never deleted; they are archived (CLAUDE.md: soft delet
  12. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-zblCwm/src/main/java/com/acme/notes/notes/NoteController.java","old_string":"import org.springframework.web.bind.annotation.PostMapping;\nimport or
  13. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-zblCwm/src/main/java/com/acme/notes/notes/NoteController.java","old_string":" @PostMapping(\"/{id}/archive\")","new_string":" @PutMapping(\"/
  14. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-zblCwm/src/test/java/com/acme/notes/NoteApiTest.java","old_string":"import static org.springframework.test.web.servlet.request.MockMvcRequestBuilde
  15. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-zblCwm/src/test/java/com/acme/notes/NoteApiTest.java","old_string":" @Test\n void missingNoteIsAnEnvelopeError() throws Exception {\n
Files changed (4)
  • src/main/java/com/acme/notes/notes/NoteController.java
  • src/main/java/com/acme/notes/notes/NoteDtos.java
  • src/main/java/com/acme/notes/notes/NoteService.java
  • src/test/java/com/acme/notes/NoteApiTest.java
Response
Now the controller:

Now add a test mirroring the existing conventions:
with run 2 1.00 11 turns · $0.186 · 42 s · error
✓ envelope-and-mapping✓ llm-conventions✓ no-try-catch✓ only-expected-files✓ skill-used
Judge reasons
  • llm-conventions — All criteria met: (1) update endpoint returns ApiResponse envelope consistent with other endpoints; (2) UpdateNote declared as record inside NoteDtos alongside CreateNote; (3) NoteController uses constructor injection with private final field, no @Autowired; (4) note-not-found error in NoteService.update() throws ApiException directly with no try/catch in controller.
Tool calls (13)
  1. Skill {"skill":"notes-conventions"}
  2. Bash {"command":"find /tmp/eval-shim-ws-update-note-endpoint-Nzgr6F -type f -name \"*.java\" | grep -i note"}
  3. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-Nzgr6F/src/main/java/com/acme/notes/notes/NoteController.java"}
  4. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-Nzgr6F/src/main/java/com/acme/notes/notes/NoteService.java"}
  5. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-Nzgr6F/src/main/java/com/acme/notes/notes/NoteDtos.java"}
  6. Read {"file_path":"/tmp/eval-shim-ws-update-note-endpoint-Nzgr6F/src/test/java/com/acme/notes/NoteApiTest.java"}
  7. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-Nzgr6F/src/main/java/com/acme/notes/notes/NoteDtos.java","old_string":" public record CreateNote(@NotBlank @Size(max = 120) String title, @Size(
  8. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-Nzgr6F/src/main/java/com/acme/notes/notes/NoteService.java","old_string":" /** Notes are never deleted; they are archived (CLAUDE.md: soft delet
  9. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-Nzgr6F/src/main/java/com/acme/notes/notes/NoteController.java","old_string":"import org.springframework.web.bind.annotation.PostMapping;\nimport or
  10. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-Nzgr6F/src/main/java/com/acme/notes/notes/NoteController.java","old_string":" @PostMapping(\"/{id}/archive\")","new_string":" @PutMapping(\"/
  11. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-Nzgr6F/src/test/java/com/acme/notes/NoteApiTest.java","old_string":"import static org.springframework.test.web.servlet.request.MockMvcRequestBuilde
  12. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-Nzgr6F/src/test/java/com/acme/notes/NoteApiTest.java","old_string":"import org.junit.jupiter.api.Test;\nimport org.springframework.beans.factory.an
  13. Edit {"replace_all":false,"file_path":"/tmp/eval-shim-ws-update-note-endpoint-Nzgr6F/src/test/java/com/acme/notes/NoteApiTest.java","old_string":" @Test\n void missingNoteIsAnEnvelopeError() throws Exception {\n
Files changed (4)
  • src/main/java/com/acme/notes/notes/NoteController.java
  • src/main/java/com/acme/notes/notes/NoteDtos.java
  • src/main/java/com/acme/notes/notes/NoteService.java
  • src/test/java/com/acme/notes/NoteApiTest.java
Response
Now I have the full picture. I'll add an `UpdateNote` DTO, a service `update` method (PUT-style, reusing the not-found/archived rules), and a controller endpoint, plus a test.

Now the controller. I'll use `@PutMapping` for the update endpoint, consistent with REST conventions (the existing archive action uses POST for a state-transition action, but a full field update fits PUT).

Now add a test mirroring the existing style.

Hover a grader chip for its type and reason. Indicators (ind) are recorded but not scored. Generated by config-drift-checker · hosted beta.