{
 "model": "claude-haiku-5-5",
 "category": "context",
 "scenarios": [
  {
   "id": "c1-handoff",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Two handoffs across 3 sessions. ARCHITECTURE.md (10,711 bytes) was finished in session 3 and names all 11 engine files, three risks and five labelled open items. I checked 79 backticked names against the source and all exist. Exit codes 0,0,0; driver.err empty; no leftovers; 38 tool-use/38 result frames paired; 0 unpaired in saved contextMessages; sessions mode 600; cost recompute diff 0.",
   "notes": "Read-before-handoff was sent at both handoffs, each after unread tool results (status frame, then engine note '[The context window is nearly full…' in saved contextMessages). No tool frames between that status and 'Writing a handoff'. Saved interrupted messages have the reply starting after a blank line ('…full.\\n\\n## Handoff', '…source.\\n\\n## Handoff'). Neither note was cut and neither has the cut-off marker. Room minus 256 was 12,963 and 4,284 tokens; the reply's stopReason is not saved. No 'not seen yet' text in either handoff .md; session/handoff.ts:152-153 sets unseenFrom to lastReply+1 so the writer sees those results as read (the writer's prompt is not saved, so this comes from the code). Stalls 1 then 2 (no file changed in sessions 1-2); continuations carry 'Progress check: 1 session' and '2 sessions', with no sub-agent wording (below 3). Binding rule leads every continuation and saved .md; <request> stays the original in both. Old sessions are saved interrupted with the handoff path in metadata.reason. Seeds carried goal, done and open work, and were accurate (10 files full, queryEngine.ts 1-514). Model behaviour: seed 1 labelled lines 218-355 facts 'in the trimmed region, unverified in this copy', and session 2 re-read 215-514; session 3 ran broad greps and three narrow reads although told not to re-verify. It did not re-read whole finished files. Cleared 4 old results in session 3 once, pairs intact. Tools 31 incl. Skill, no skill block, no Guillotine or DSH effects."
  },
  {
   "id": "c2-compact",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Asked once (contextFull 1), compacted 4 times mid-turn with no API error, and the turn completed. ARCHITECTURE.md (12,767 bytes) covers all 11 modules, data flow and three risks; 87 backticked names checked against source, all exist. Exit 0, driver.err empty, no leftovers, 60/60 frames paired, 0 unpaired saved blocks, compaction_count 4, cost recompute diff 0.",
   "notes": "The user's request stays word for word as contextMessages[0], with the summary next to it holding what is left, files, line ranges and state. Heavy re-reading (FileRead 29, Bash 25) came from a 60K budget that cannot hold the 108 KB of source plus reserves. The clip path cut fresh parallel results to about 1,500-1,800 characters, and old-result clearing then removed clipped first-round results. This is the designed clip path; I read engine/compact.ts and microcompact.ts. The cut notes are accurate ('Truncated for context management… call FileRead with offset=N'). Model misread: Haiku called this a FileRead limit of about 1,450 characters, tested Bash for a larger limit, and carried the false claim into its summary. Flagged as a cost observation for the user, not a defect: mid-turn compaction clips results the model never read, because read-before-handoff has no compaction equivalent. Compared with r17f: 4 compactions against 3 and $0.052 against $0.029, on identical engine code (only the 5 security/subprocess files differ, confirmed by diff), so this is model variance. cacheInvalidations 6 equals 4 compactions plus 2 clears."
  },
  {
   "id": "c3-stop-then-continue",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Stop ended turn 1 with a clear '93% full' error (47,194+9,000 of 60,000) and it was saved as interrupted. /compact shrank 47,272 to 16,894 with the archive kept in compactArchives. The continuation read only queryEngine.ts 515-end and wrote ARCHITECTURE.md (12,204 bytes, all 11 modules, 106 names checked). No 400s, exit 0, no leftovers, frames paired, cost recompute diff 0.",
   "notes": "Saved compacted text has 3 parts labelled 1, 2 and 3 of 3, and the user's goal is in each. Part prompts are not saved, so I reproduced them by running the snapshot's summarizeContext on the saved archive with a fake client under a scratch HOME (reserve approximated; probe since removed). It gave 3 parts, matching the live count. Each has 'Part i of 3' and the user's request, not the stop marker. Results are labelled '[tool result of FileRead <full path ending in the file name>]'. Part 2 ends with '[this block continues in the next part]' and part 3 opens with '[continued from the previous part: … queryEngine.ts]'. Live summaries agree (part 2 ends at line 314, part 3 starts at 315). The results carry the unseen heading because no reply followed them. abort_count 1 is by design (queryEngine.ts:1013 counts ContextStopped as a user stop; same in r17f). The marker '[The user stopped this turn…]' is kept beside the summary. Model note: its final reply honestly says lines 315-514 rely on notes."
  },
  {
   "id": "c4-handoff-chain",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Four handoffs across 4 sessions; the driver followed 3 (hops<3 in driver10b.ts), so the fourth was not continued. SUMMARIES.md is 60 lines: all 11 engine/ entries plus engine2/microcompact, each file once with no duplicates. engine2/ is otherwise unfinished only because of the harness limit, and the final seed states the exact state and next steps. Exit codes 0, driver.err empty, no leftovers, 36/36 frames paired, 0 unpaired saved blocks, cost recompute diff 0.",
   "notes": "Read-before-handoff was sent at all 4 handoffs after unread results, with no tool frames in it, and each saved reply starts after a blank line. No note cut (a grep hit for 'cut off at' in session 2 is the model's own words about a read, not the marker; no '[This note was cut off' anywhere). Stalls: none, none (Bash changes tracked in outputs), 1, none. Only continuation 4 has 'Progress check: 1 session', correct and with no sub-agent wording. Every continuation opens with the binding rule and quotes the ORIGINAL request; no loop. No 'not seen yet' text in any .md (writer's view from code, as in c1). No handoff .md mentions BrowserVerify and no skill block appears, so D17-9 was not stressed here. Model behaviour: session 3 wrote nothing while reading queryEngine.ts chunks; session 4 wrote those entries from notes and fixed a 4-line entry itself. 'Cleared 2' in session 1 is harmless."
  },
  {
   "id": "c5-huge-log",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "REPORT.md verified independently with grep/sort/uniq: W20 69,326, W10 46,455, E404 18,730, E500 11,674, E200 6,908; first E500 00:00:15. Last E500 is 20:26:37 by file position and 23:59:59 by value (the log's clock wraps), and the report gives both. Exit 0, no leftovers, frames paired, cost recompute diff 0.",
   "notes": "The model used only awk/grep Bash with small outputs, so FileRead on the 12.5 MB file and persisted large Bash output were not exercised in this run. Context stayed small; cache read ratio 0.943."
  },
  {
   "id": "c6-many-files",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "All 150 notes read (150 distinct FileRead paths, batches of 30-40). I verified INDEX.md: 150 entries, each once, none missing, every entry in the right Topic section (n mod 7). Exit 0, no leftovers, 153/153 frames paired, cost recompute diff 0.",
   "notes": "No context-full at the 120K budget (47,481 used). toolHistory holds 100 of 153 calls, the designed cap. Config ran High effort as the brief says, although scenarios.json says medium. Cache read ratio 0.879."
  },
  {
   "id": "c7-multi-turn-cache",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Four turns completed. JSDoc was added to all methods, and count() with a test. My own npm test gives 6 tests, 1 pass, 5 fail, matching the model's report; the 5 failures are seed bugs it was not asked to fix and it said so. Exit 0, no leftovers, frames paired, cost recompute diff 0.",
   "notes": "Cache read ratio 0.952 overall; last request read 22,736 of 23,042 input tokens; uncached input across all 13 requests is 34 tokens, so the prefix stays cached across turns with no sign of an invalidating system prompt. Per-request usage is not saved, so the rise across turns is inferred from totals and lastUsage. The one failed Bash was `npm test | grep` with no match (exit 1), correctly state failed. learned/patterns.json: `npm test` runs 5, successes 0, broken 0, correct for five calls ending in tail or grep (status unknown); under the run's own HOME."
  },
  {
   "id": "j6-long-session",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "All 6 user messages completed across 2 sessions, with one handoff during message 5 and a 7th turn for the continuation. My own npm test: 33 tests, 33 pass. src/cli.js, tests and README are present, and the CLI prints usage. The final summary recalls all earlier work and separates this session from the previous one. Exit codes 0,0; driver.err empty; no leftovers; 33/33 frames paired; 0 unpaired saved blocks; cost recompute diff 0.",
   "notes": "Handoff at 64,703+12,288 of 90,000 (85.6%) after unread results: read-before-handoff sent, no tools, reply after a blank line, not cut. Stalls none because files changed. Continuation leads with the binding rule, <request> is message 5's words; message 6 was then sent normally in the new session. The seed lists all earlier requests, done work and open steps, and the first session is saved interrupted with the handoff path. Heartbeat statuses at 20s, 35s and 50s while waiting. Cache ratio 0.913. learned/patterns.json: `npm test` runs 5 and `node --test test/cli.test.js` runs 1, all successes 0, matching the 6 piped calls; not trusted, so not in the system prompt. Model changed seed behaviour to fix failing tests when asked and listed the duplicate-SKU consequence for the user."
  }
 ]
}