{
  "model": "claude-haiku-5-5",
  "category": "context",
  "scenarios": [
    {
      "id": "c1-handoff",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Completed ARCHITECTURE.md after three handoffs and three explore Agent calls; independently checked all 11 module sections, data flow and three risks against the unchanged seed sources.",
      "notes": "Four saved sessions and four exit-0 completions; frames and saved tool calls/results pair, driver.err empty, recorded leftovers empty. All sessions advertise exactly 30 tools with the six-skill catalog; no DSH tools/directory or team-memory leakage observed. All continuations preserve the original request and binding rule; handoff files contain the rule, summaries and source session IDs; old turns are interrupted. Read-before-handoff ran three times, with no tools, a blank line in each saved assistant message, no truncation marker and no subsequently unread result. Frozen source plus context-full figures imply caps 12734/10504/17438 tokens. Stalls 1/2/3 are correct; Agent instruction appears only at 3, followed by actual explore calls. Cleared three old results without broken pairing. The model repeatedly reread finished material; the second writer incorrectly said the original request was absent, but the continuation separately retained it verbatim. A Grep against absent fixture memory/ genuinely ran and failed, correctly recorded failed. One risk remains explicitly unverified because this fixture excludes imported modules; no runnable project build exists. Recomputed 41-request offline estimate $0.101188405 matches result.json; cache read 699533/1012117. This is not a provider invoice. Evidence: review-context-audit.json, review-context-handoff.json, review-context-artifact.json."
    },
    {
      "id": "c2-compact",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Completed ARCHITECTURE.md covering all 11 unchanged fixture modules, data flow and three risks after three in-turn compactions.",
      "notes": "One context-full question, three successful compactions and two old-result clearings (9 and 7); subsequent requests completed with no provider 400 or orphan tool blocks. Saved summary retains the verbatim request, finished reads, missing ranges and next steps; final context retains valid tool pairs. Manual multipart /compact is not exercised here. The model misread transcript trimming as incomplete parallel reads and repeated work; an atomic 12-edit batch failed at edit 11 with no edits applied, then was corrected. This is artifact/model behavior, with a correctly failed executed tool. Driver exit0, empty driver.err, no unparsed frames or recorded leftovers, 30 tools and full Skill catalog, no DSH artifacts. Recomputed 20-request offline estimate $0.04047406 matches; cache read 389521/552012. Documentation fixture has no runnable build. Evidence: review-context-audit.json and review-context-artifact.json."
    },
    {
      "id": "c3-stop-then-continue",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "The requested stop persisted an interrupted turn; /compact preserved a three-part summary and archived original wire history; continuation completed an 11-module ARCHITECTURE.md.",
      "notes": "The 92% stop message agrees with input 46444 plus reserve 9000 in a 60000 budget. The error is expected, followed by two turn-complete frames, no API 400, driver exit0 and valid saved/frame tool pairs. Saved compacted text has Part 1/2/3 goals, file/range facts and the hooks block continuation. An independent offline stub probe using the saved archive, frozen summarizeContext and explicit reserve 20000 passed: each of three requests has Part i of 3 and the true user goal, tool-result labels retain tool/path, the cut hooks block ends with the continuation marker and the next part names it, and a long synthetic path retains its filename. No model/network request was made. Final probe exit0; two earlier reviewer-script assertion errors (omitted reserve and an over-specific newline expectation) are preserved in separate review logs. Empty driver.err, recorded leftovers empty, 30 tools/full Skill catalog, no DSH artifacts. Recomputed 9-request estimate $0.01625146 matches; cache read 116191/176638. Evidence: review-context-compact-probe.json/log, review-context-audit.json, review-context-artifact.json."
    },
    {
      "id": "c4-handoff-chain",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Preserved seven unique five-bullet summaries across four sessions and four handoffs; the requested 22-file task remains incomplete because the driver stops after three continuations.",
      "notes": "Independently counted 7 entries/35 bullets, no duplicates, 15 missing entries and byte-identical engine/engine2 seed sources. All three actual continuations retain the original request, binding rule, completed entries and open work; every old turn is interrupted and its handoff saved. Stalls 0/0/1/2 agree with changes in the first two sessions and none in the last two; no premature Agent instruction. Read-before-handoff occurs in sessions 1/2/4, runs no tools, starts after a blank line, has no cut marker and makes results seen by the writer; inferred caps 9135/3660/5335. Session 3 has measured input 43101 above its 40000 window, so skipping that reply is correct; its handoff explicitly marks latest results unseen. The model repeatedly reread material and sometimes confused transcript trimming with missing reads despite the writer instructions. driver10b.ts:141 limits following handoffs to three; the fourth handoff was saved but not followed. Four exit0 processes, empty driver.err, paired frames/sessions, 30 tools/full Skill catalogs, no DSH artifacts or recorded leftovers. Recomputed 26-request estimate $0.03532376 matches; cache read 415431/559736. Evidence: review-context-handoff.json, review-context-artifact.json, review-context-audit.json."
    },
    {
      "id": "c5-huge-log",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Independently scanned all 160000 lines/12573109 bytes: top-five counts and E500 timestamps in REPORT.md are correct.",
      "notes": "Counts are W20 69326, W10 46455, E404 18730, E500 11674, E200 6908. First/last E500 in file order are 00:00:15/20:26:37; chronological maximum is 23:59:59, correctly distinguished because timestamps wrap. REPORT.md gives the final E500 line as 159997 instead of 159998: a minor model-authored annotation error, not a G-Code defect. The model used bounded Bash aggregations, so FileRead overflow and large-output persistence were not exercised; no context-full event or blowup. Driver exit0, empty driver.err, valid pairs, 30 tools/full Skill catalog, no DSH artifacts or recorded leftovers. Recomputed 6-request estimate $0.00274073 matches; cache read 78358/82698. Evidence: review-context-audit.json."
    },
    {
      "id": "c6-many-files",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Independently verified INDEX.md has exactly 150 unique entries, each under the correct Topic with text matching its note file.",
      "notes": "The actual Bash loop reads all three lines of every note and generates the index; all 150 IDs, link targets, topics and bodies match, with no missing or duplicate entry. Link label and destination are one entry. The efficient Bash approach used only eight parent calls rather than 150 FileReads; no context-full event, so compaction was not exercised here. Driver exit0, empty driver.err, valid pairs, 30 tools/full Skill catalog, no DSH artifacts or recorded leftovers. Recomputed 6-request estimate $0.00460021 matches; cache read 102581/118347. Evidence: review-context-audit.json."
    },
    {
      "id": "c7-multi-turn-cache",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Added JSDoc and count() with a passing distinct-SKU test; independently ran the project and original seed tests using private Node v24.22.0.",
      "notes": "Project node --test exits 1 with 1 pass/5 failures; an isolated seed copy exits 1 with the same five original failures. The new count test passes and the model accurately disclosed the existing failures, so this is not a product failure. Four turns complete in one persisted session, valid pairs, no cache invalidations. Aggregate cache read is 208265/218797 (95.1864%); last request reads 21467/21890 (98.0676%). Per-request/per-turn usage and historical system snapshots are not logged, so a rising curve or positive cache read specifically on request 2 cannot be proved directly; evidence does not support a changing-system cache defect. Driver exit0, empty driver.err, 30 tools/full Skill catalog, no DSH artifacts or recorded leftovers. Recomputed 13-request estimate $0.0059448 matches. Tests used fresh HOME/USERPROFILE and scenario-local TMPDIR, no installs or servers; subprocesses exited and tmp was empty. Evidence: review-context-tests.json/log, review-context-seed-tests.log and review-context-audit.json."
    },
    {
      "id": "j6-long-session",
      "status": "pass",
      "product_defects": [],
      "task_outcome": "Completed six turns of validation, persistent inventory CLI, README and fixes; independent private Node v24.22.0 test run passed all 42 tests with exit0.",
      "notes": "One persisted session retains all six user/assistant turns; final summary accurately recalls earlier validation, CLI, tests, documentation and remaining artifact limitations. No handoff was needed. All executed calls and saved wire pairs are valid. Aggregate cache read 856195/897175 (95.4323%); last request 51590/52310 (98.6236%), no cache invalidations. Exact request-2/per-turn ratios are unavailable from these logs. Driver exit0, empty driver.err, 30 tools/full Skill catalog, no DSH artifacts or recorded leftovers. Recomputed 26-request offline estimate $0.03090335 matches. Tests ran with fresh HOME/USERPROFILE and local TMPDIR; all temporary test directories were cleaned and all owned processes exited; no listener was started. All eight context costs are complete saved-usage estimates, not bills. This DSH-off CLI review does not establish native desktop/PTY/DSH coverage. Evidence: review-context-tests.json/log and review-context-audit.json."
    }
  ]
}
