{
  "model": "claude-haiku-5-5",
  "category": "edit",
  "scenarios": [
    {
      "id": "b1-inventory-fix",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Fixed all five seeded inventory/report defects and added exportCsv with four CSV tests; independent Node 24.22.0 node --test passed 9/9, exit 0. Original test bytes unchanged; all three Changes entries, original baselines and final snapshots verified.",
      "notes": "12 saved tool pairs. Initial test failures were visible inside a shell command that ultimately exited 0; successful edits and reruns completed normally. Cost recomputed $0.005300715. Shared checks for all 11 scenarios: valid sessions, complete turn endings, no unparsed/error frames, correct tool-result pairing and terminal states, driver exit 0, empty driver.err and recorded leftovers, positive cache reads, 30 advertised tools with all six Skill catalog descriptions, no AskUserQuestion/DeferredToolCall/DSH tools or run-home DSH directory. All saved usage totals are complete and all recorded costs/cache ratios recompute exactly under frozen pricing, including long-prompt rates; these are estimates, not provider invoices. Evidence is in each scenario/review16b-independent/. No native desktop, PTY or DSH-on coverage is claimed; no reviewer application/model calls or dependency installs."
    },
    {
      "id": "b2-inventory-approvals",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Independent original tests passed 5/5, exit 0; two source Changes baselines/final versions match and test bytes are unchanged.",
      "notes": "Five unique approval requestIds: three Bash calls and two FileEdit calls, each preceding its execution. The driver answers by matching requestId with allowed:true; outbound approval bytes are not separately logged. The pre-fix npm test ran and failed, correctly saved failed; its later post-edit rerun is a new one-shot approval, not a duplicate prompt for the same call. No remembered grant was requested. Nine saved pairs and one turn-complete; cost estimate $0.003269785."
    },
    {
      "id": "b3-inventory-deny",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "One Bash approval was declined; no tool-use frame or runningAt exists, the terminal state remains denied and the result says Nothing was run. All four seeded nonmetadata files remain byte-identical; no Changes entries. Both turns completed.",
      "notes": "The second answer accurately explains the denial in two sentences. One saved tool-use/result pair is preserved although the protocol correctly emits only the denial result. Independent tests ran only in a reviewer copy and failed all five original tests (exit 1), as expected for untouched seed bugs. AppleDouble sidecars are merged by seed cp -R, not deleted by the model. Cost estimate $0.00105746."
    },
    {
      "id": "b4-js-to-ts",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Strict TypeScript conversion completed; independent TypeScript 7.0.2 --noEmit exited 0 and the original five tests passed with Node 24.22.0 and installed tsx 4.23.15. Eight Changes entries include both deleted .js originals, absent baselines for newly created .ts files, and intact version history.",
      "notes": "Two FileWrite calls actually entered tool.call and returned the read-before-overwrite error, correctly saved failed; FileRead then successful writes recovered. All 16 saved pairs match. Reviewer tsx CLI initially failed EINVAL because the isolated TMPDIR made its Unix socket path too long; that log is retained. Running the same tests through node --import tsx --test passed 5/5 without its CLI IPC socket. No dependencies changed. Cost estimate $0.00622544."
    },
    {
      "id": "b5-python-lib",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Fixed quantity totals, percent conversion and discount bounds; independent python unittest discovery passed 3/3, exit 0. shop/cart.py baseline matches seeded bytes; test_cart.py is unchanged.",
      "notes": "The seed lacked tests/__init__.py, so initial discovery could not import tests. The model added the empty package marker and fixed only shop/cart.py; both Changes entries are correct. Shell pipelines explain why the initial child failure was not a failed Bash tool result. Eleven saved pairs; cost estimate $0.003275305."
    },
    {
      "id": "b6-rename-across-files",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "All 16 source files use getDisplayName with no old-name references. Independently ran the reviewed script: all modules import, all 15 callers return Hi Ada Lovelace, exit 0.",
      "notes": "Seventeen unique Changes entries cover 16 source edits plus the new verification script; all seeded baselines and final versions match, with no lost edits. Twenty-one saved pairs; cost estimate $0.004855845."
    },
    {
      "id": "b7-notebook",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Two NotebookEdit calls appended the requested code and explanation while preserving the original cell contents. Notebook remains nbformat 4 with four cells; independently executed reviewed code and obtained mean 7.25 and max 13, exit 0.",
      "notes": "The single Changes entry preserves the original JSON bytes and both edited versions. Generated cell IDs are valid. Seven saved pairs and no tool failures; cost estimate $0.002935525."
    },
    {
      "id": "b8-large-file-edit",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Independent full-file comparison confirms only fn100/fn1500/fn2999 multiplication, fn42 deletion and a JSDoc above fn0. Reviewed verification script parsed the module and checked all 2999 exported functions across five inputs, exit 0.",
      "notes": "Original 3000-line baseline, both big.js edit versions, latest content and separate verification-script Changes entry all match. Eleven saved pairs; no lost or collateral source edits. Cost estimate $0.006846095."
    },
    {
      "id": "j3-refactor-legacy",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Split the three largest seeded source components (Subjects, Contact, Home). Independently preserved the original bundling flags: seed exits 1 with 19 errors and final exits 1 with 28, both caused by the same nine undeclared dependencies. Supplementary checks using only existing live-created dependencies passed bundling and 11/11 normalized DOM/form-state comparisons; both typechecks retain the identical five seeded CSS/font declaration errors.",
      "notes": "No clean original build is claimed. Additional bundle uses existing /tmp/deps-8965 dependencies and --conditions=style, without widening the original external list. TypeScript 7.0.2, esbuild 0.28.2, React 19.3.1 and jsdom 30.1.3 versions/source are recorded. The first reviewer CJS harness failed on seed NumberFlow interop under type:module; its failure is preserved. Audited live harness was then run against unchanged source copies with the same Babel/CJS test interpretation used by the live comparison. Context7 lacked exact 0.28.2; exact upstream runtime confirms the interop rule: [esbuild v0.28.2 runtime](https://raw.githubusercontent.com/evanw/esbuild/v0.28.2/internal/runtime/runtime.go). This supplementary DOM check normalizes IDs/animation styles and does not establish browser layout or animation fidelity. All 48 tool results/states are complete: shell pipelines/follow-up commands masked child failures, including eight explicitly printed nonzero child exit codes and four harness-failure transcripts (overlapping). Failure guard therefore did not fire; no failed/refused call was mislabeled. Thirty-one unique Changes entries are intact, including Subjects.tsx deletion with original baseline; deleted binary AppleDouble ._Subjects.tsx has no baseline by outputs.ts:443 policy. Cost recomputes $0.289654815 using 22 long-prompt requests. See original-build-*.log, typecheck-*.log, dom-comparison.json and live-internal-failures.json."
    },
    {
      "id": "j4-docs-from-code",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Created 11 module API pages plus README. Independent source/document checks found all 31 exported functions/classes documented, signatures consistent and all links resolving; engine source bytes remain unchanged.",
      "notes": "Twelve unique new-file Changes entries and 28 saved tool pairs verified. calculateContextBudget documents options? rather than its source default = {}, an equivalent optional call signature. The fixture contains historical engine wording; these are docs of the supplied seed, not claims about the current product. There is no executable documentation build. Cost estimate $0.018732285."
    },
    {
      "id": "j5-git-project",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Independent original tests passed 5/5, exit 0. Fix commit a329d04ec1b4f0e83a2e351d2b0ce083ac0df73a changes only the two source files; working tree is clean, original tests unchanged and git fsck exits 0.",
      "notes": "Two Changes baselines match HEAD^ bytes and final snapshots match HEAD. The live tool evidence includes the requested diff stat and successful commit; independent git log confirms init followed by the fix. Ten saved pairs; cost estimate $0.00360759. Reviewer git commands used optional locks disabled and did not alter repository state."
    }
  ]
}
