{
 "model": "claude-haiku-5-5",
 "category": "edit",
 "scenarios": [
  {
   "id": "b1-inventory-fix",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Fixed the 5 bugs in src/inventory.js and src/report.js, and added exportCsv with test/report.test.js. I ran npm test: 9/9 pass, and test/inventory.test.js is unchanged from the seed. The FileEdit baselines have the same sha256 as the seed files (inventory.js 300ad1e8…, report.js 9b1a8507…). New test file has an absent baseline. No duplicate outputs.",
   "notes": "Exit 0, 1 turn-complete, 11/11 tool calls paired, all complete. Recomputed cost $0.00522726 matches; cache-read ratio 0.92. learned/patterns.json key 'npm test' shows runs 2, successes 0. That is correct under D17-1: both runs were `npm test 2>&1 | tail -N`, so the exit status is unknown. Leftovers empty, driver.err empty."
  },
  {
   "id": "b2-inventory-approvals",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Tests were run, bugs in src/ were fixed and tests were left alone. I ran npm test: 5/5 pass. Baselines match the seed bytes.",
   "notes": "5 tool-approval frames for 5 different calls (3 Bash, 2 FileEdit). Each was approved by requestId and the call ran right after; no hang and no call was asked twice. The 3 FileReads asked for no approval. Recomputed cost $0.00361508 matches. The learned key keeps the leading `cd \"…\" &&`. That is by design: the key is the command up to its last check. successes 0 because of `| tail`, which is correct."
  },
  {
   "id": "b3-inventory-deny",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "The one Bash call was denied and nothing ran. The project matches seeds/edit-existing exactly (diff -r is empty, npm test still 0/5 as seeded). The second turn explained the denial correctly.",
   "notes": "toolHistory state is 'denied'. The frame error reads 'You declined this Bash call. Nothing was run.' and the tool_result is paired with is_error. There was no tool-use frame for the denied call, which is by design. Nothing was asked again. Both turns ended with turn-complete. No outputs and no learned file. The model chose to ask the user rather than try file tools."
  },
  {
   "id": "b4-js-to-ts",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Converted to src/*.ts with strict types, a strict tsconfig, typescript/tsx/@types/node as dev dependencies, and a RangeError for an empty array in mean. I ran `tsc --noEmit` (clean) and npm test (node --import tsx --test): 5/5 pass.",
   "notes": "The Bash `mv` rename is recorded correctly in Changes: math.js and stats.js are marked deleted with their 97-byte baselines, and the .ts files have absent baselines and 2 versions. package.json keeps its original 33-byte baseline with 2 versions. Two FileWrite 'exists but has not been read' errors came from the model writing over its own mv'd files before reading them. These are correct 'failed' states (the check is in call()); the model then read the files and retried. The model also wrote /tmp/pkg-before.json outside the project. Cost matches."
  },
  {
   "id": "b5-python-lib",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Fixed shop/cart.py (qty multiply, percent discount, ValueError when out of bounds). I ran `python3 -m unittest discover -s tests -t .`: OK, 3 tests. test_cart.py was not edited.",
   "notes": "The model added an empty tests/__init__.py, which shows in outputs as a create. The cart.py baseline is the 118-byte seed. Learned keys follow the 'up to the last check' rule, and successes are 0 because of `; echo`, which is correct. Cost matches."
  },
  {
   "id": "b6-rename-across-files",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Renamed getUserName to getDisplayName in all 16 files. grep finds no old name in src/, and I re-ran check-imports.mjs: 16/16 modules import and every f<N> works.",
   "notes": "Outputs list all 16 edited files once each, plus check-imports.mjs. Every baseline snapshot contains getUserName (the original bytes). There are no duplicates. 37/37 calls paired. Cost matches."
  },
  {
   "id": "b7-notebook",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "NotebookEdit inserted a code cell (mean/max) and then a markdown cell after the existing cells, editing the file in place. I validated it with nbformat.validate (installed by the model in the run HOME): valid, nbformat 4.5, 4 cells in the right order.",
   "notes": "The notebook has one outputs entry with its original 242-byte baseline and 2 versions. Cell ids were added, which nbformat 4.5 allows. The model pip-installed nbformat into the run HOME to validate. Cost matches."
  },
  {
   "id": "b8-large-file-edit",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "A unified diff of big.js against its baseline shows only the requested changes: JSDoc above fn0, fn42 removed, fn100/fn1500/fn2999 changed to multiply. I imported the module: fn100(3)=300, fn1500(2)=3000, fn2999(2)=5998, fn42 absent, 2999 exports.",
   "notes": "The baseline holds the original 3000-line file (170,780 bytes). The model used Grep and one FileEdit with 5 edits (an edits array) instead of reading the whole file. It also wrote scratch copies in /tmp (/tmp/big.js.orig, /tmp/vcheck, which it removed). Cost matches."
  },
  {
   "id": "j3-refactor-legacy",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Split the 3 largest files (Subjects.tsx 475 lines, Contact.tsx 348, Home.tsx 319) into 25 new files in the same folders. The originals are now thin barrels or composers. I rebuilt the bundle with esbuild using the model's /tmp/deps-8965 install plus --conditions=style: it builds. I rebuilt and ran the model's server-render harness from the seed src and the project src: all 197 rendered snapshots are byte-identical, so behaviour is unchanged.",
   "notes": "Context management behaved as designed. The cost comes from the model's own choices, not from G-Code. Evidence: the window is 1M tokens and compaction (and the clearing tried before it) starts at 85%, about 850K (snapshot10w/src/engine/context.ts:102; queryEngine.ts ~1391). The final contextStatus is used 159,126 / 1,000,000 (16%). Clearing runs only when budget.needsCompaction is set, so no clearing, context-full, compaction or handoff should happen here, and none did (systems [], contextFull 0, compactions 0). The 100K figure is only the Haiku long-prompt price card (shared/pricing.ts), not a context trigger. The model did not re-read files: there were only 5 FileReads, of 5 different files. The extra requests come from mostly one tool call per reply (57 calls in 53 replies) and much more self-checking than attempt a's 22 calls: an npm deps install in /tmp, a tsc before/after comparison, a server-render harness with 2 fix-ups, and BrowserPlan, BrowserPlanCheck and 2 BrowserVerify runs (both passed). The prompt also carries the turn's signed thinking blocks (about 245K chars of continuation), which the Anthropic API requires inside a tool loop. Cache read ratio is 0.974. The recomputed cost with the long-prompt split (36 requests) is $0.4143115, which matches. Failures: 4 in total and the failure guard never fired, which is correct: they were all different and successes came in between. Two Bash calls (server-render harness errors) are 'failed'. BrowserPlan with an unknown key 'kind' is 'refused' (invalid input, no tool-use frame, by design). The BrowserPlan entry error is 'failed' (thrown in call(), single 'Error:'). Changes: the Subjects/Home/Contact baselines have the seed sha256, new files have absent baselines, there are no duplicates, and home-sections-8965.json (written by BrowserPlan, deleted by Bash) is marked deleted. The model's .gcode/verification/site-new copy is not listed because outputs.ts SKIPPED excludes .gcode from listings, which is by design. Leftovers are empty. The model left scratch files in /tmp (deps-8965, src-orig-8965, bundle-8965.sh). The exact task command without --conditions=style fails on the seed too (`@import \"tailwindcss\"`), as the model found."
  },
  {
   "id": "j4-docs-from-code",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Wrote docs/<name>.md for all 11 engine/*.ts files and a docs/README.md that links each one. I checked with a script: every regex-detected export is named in its doc. engine/ is unchanged from the seed.",
   "notes": "12 FileWrite creates with absent baselines and no duplicates. 29/29 calls paired. Context peaked at 83K tokens. Cache-read ratio 0.81 (large writes). Cost matches."
  },
  {
   "id": "j5-git-project",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Fixed src/inventory.js and src/report.js. Commit 5730ee5 'Fix inventory accounting, stock checks, and value formatting' sits on top of init 4c4cbec. The working tree is clean and git fsck is clean. I ran npm test: 5/5 pass. git diff --stat was shown.",
   "notes": "The baselines (748/147 bytes) are the seed bytes, which are also the init commit's contents. The tests are unchanged. learned 'npm test' shows runs 3, successes 0, which is correct: every run was piped through `| tail` or `| grep`. Cost matches."
  }
 ]
}