{
 "model": "claude-haiku-5-5",
 "category": "context",
 "scenarios": [
  {
   "id": "c1-handoff",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Three sessions and two handoffs. ARCHITECTURE.md was written in session 3 and covers all 11 modules; all 72 quoted names exist in engine/. Exit codes were 0, driver.err was empty, frames paired 27/27, and saved sessions paired with cache reads > 0 in each.",
   "notes": "Both handoffs followed unread tool results, and G-Code sent the read-before-handoff request each time. No tool frames appeared between that status and the writer status. Replies of 8,261 and 5,683 chars each started after a blank line, ran no tools and were not cut off. Neither saved handoff file has an unseen-results heading. Stalls were 1 then 2. Continuations carried 'Progress check: 1 session' and '2 sessions', with no sub-agent ask. The binding rule led each continuation, and the chained <request> kept the original words. Both old sessions were saved interrupted and linked to their handoff paths. Model behaviour, not a defect: the seeds were accurate, but session 2 re-read queryEngine 1-514 and session 3 re-read 515-967. The second seed said the 710-890 middle was elided in the writer's copy, which may have prompted the session 3 re-read. No skill was loaded and tools were 31. Cost matched the recompute."
  },
  {
   "id": "c2-compact",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "One turn: one context-full question, 3 compactions, 1 clearing, no API errors. ARCHITECTURE.md covers 11 modules, and all 101 quoted names exist in the source. Frames paired 46/46 and contextMessages paired.",
   "notes": "The request stayed pinned first in contextMessages, and the summaries kept file coverage by line range. Compaction clipped the 11-read batch that triggered it, so the model read gaps again, 42 FileReads in all; this inefficiency comes from the designed clip and the model's batching, and the task still finished. The summary's 'batched reads truncated to about 1,400 chars' is the model's inference. The model also briefly misread cleared results as never read and corrected itself. The saved reply's claim that queryEngine 643-661 is covered by notes was the model's statement; I did not check it line by line. Compaction was measured at 62,735 estimated tokens, above the 60,000 budget, before a request was sent; no request failed."
  },
  {
   "id": "c3-stop-then-continue",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "The 'stop' choice ended turn 1 with a clear 92% message and saved it interrupted. /compact went 46,877 to 17,820 tokens, and the continuation read queryEngine 515-967 and wrote ARCHITECTURE.md. All 11 modules are covered and all 124 quoted names exist. No 400 errors; exit code 0.",
   "notes": "I rebuilt the manual compaction requests offline by running the snapshot's summarizeContext on the saved archive with a stub client. It produced 3 parts, each with 'Part i of 3' and the user's real request rather than the engine's interruption note. Tool results were labelled with tool name and full path, and long paths kept the file name. Part 2 ended with '[this block continues in the next part]', and part 3 opened with a continued-from label naming hooks.ts. The saved text agrees: 'Part 2 of 3' and hooks.ts lines 1-186 then 187-274. Model slips, not defects: part 1 said 'treat these reads as attempted', and part 2 listed files to read that part 3 covered, despite the preface. The continuation was not misled; its one wrong guess, a hooks.ts offset past end of file, got a clear note. Listener port 8990 was clean and leftovers were empty."
  },
  {
   "id": "c4-handoff-chain",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Four sessions and 4 handoffs, stopping at the driver's cap of 3 follow-ups. SUMMARIES.md holds 35 lines, 7 engine/ blocks with no duplicates; the task is unfinished because of that cap and the model's re-reads, not a G-Code fault. Frames paired 30/30; the one failed Bash was `ls SUMMARIES.md` on a missing file.",
   "notes": "All 4 handoffs had the read-before request, no tools in the reply, no cut-off, blank-line junctions and no unseen heading. Stall frames were absent, absent, 1, 2, matching file outputs in sessions 1-2 and none in 3-4. Only the last continuation sent shows 'Progress check: 1 session'. A count of 3 and its sub-agent ask were not reached. Every continuation led with the binding rule and kept the original request in <request>. Seeds were accurate and carried facts for the read-but-unsummarised files. Sessions 3 and 4 still re-read systemPrompt, thinking and toolRunner plus part of queryEngine, then filled the 40K window before appending. That is a model choice against the seed's 'do not redo', recorded for the user as in 10h. Handoff files all carry the binding rule. Saved sessions were valid, with cache reads > 0."
  },
  {
   "id": "c5-huge-log",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "REPORT.md is correct by my own grep/awk: W20 69,326; W10 46,455; E404 18,730; E500 11,674; E200 6,908; E500 first 00:00:15, last 20:26:37 in file order. Final context was 16,774 tokens, with cache ratio 0.94.",
   "notes": "The 12.5 MB file was never read with FileRead. The model used short Bash pipelines, so large-output truncation and paging were not exercised here. It also noticed the timestamps wrap at line 86,401 and reported both the file-order last and the clock maximum. Leftovers were empty."
  },
  {
   "id": "c6-many-files",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "notes/INDEX.md has 150 entries and 150 distinct notes. I checked each entry's topic against its file's Topic line, with no mismatches. Frames paired 153/153, with no context-full (47,390 of 120,000).",
   "notes": "The index went inside notes/ rather than the project root; the model said so, and the request did not specify a location. Saved toolHistory holds 100 of 153 entries because of the designed cap (shared/runtimeEvents.ts:18). scenarios.json says medium effort, but config and controls ran High, which the task statement allows. Cache ratio was 0.886."
  },
  {
   "id": "c7-multi-turn-cache",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Four turns completed with JSDoc and count() plus a test added. I ran npm test: 6 tests, 1 pass, 5 fail. The new count test passes, and the model reported this result accurately.",
   "notes": "The failures are in the seed's own tests; the task did not ask to fix them. Cache: input 192,075, read 181,658 (0.946), write 10,387 across 11 requests, 4 turns. Per-request usage is not logged. A write total this small with almost no uncached input means reads must have started by request 2, and the prompt prefix stayed stable. patterns.json matches the commands: `npm test` runs 2, successes 0 (status unknown after a pipe), and quoted `|` in grep no longer split the command. Minor note, not a failure: a multi-check command gives the odd key `npm test 2>&1 | grep -E \"…\" ; npm test`, because redirections are stripped only from the last check's part. That key had 0 successes, so it is never shown."
  },
  {
   "id": "j6-long-session",
   "status": "fail",
   "product_defects": [
    {
     "severity": "low",
     "title": "Learned patterns count a working `npm test` as broken when test output merely contains phrases such as 'unknown command' or 'unknown option'",
     "evidence": "runs-haiku-r17c/j6-long-session/home/.gcode/projects/*/learned/patterns.json shows key 'npm test', broken 1, runs 5, although npm test ran all 5 times. log.jsonl frames 126-127: `npm test 2>&1 | grep -E ...` output has passing titles 'an unknown command is a usage error' and 'an unknown option is a usage error'. Replaying the log through the snapshot's notePatternOutcome moved broken from 0 to 1 on exactly that call.",
     "location": "snapshot10x/src/memory/learnedPatterns.ts:146 (BROKEN defined at :59)",
     "repro": "In a project whose test titles or output contain 'unknown command', run `npm test 2>&1 | grep ...` or `| tail`. Status is 'unknown', so BROKEN.test over the whole output adds a broken count. Failing runs (isError) get the same scan. Confidence falls from 0.50 to 0.33 at one count, and at 5 counts with no successes it reaches 0.14, below FORGET 0.15, so the pattern is dropped. A real but small effect: nothing in this run's turns changed, and the pattern would not have been shown anyway with 0 successes."
    }
   ],
   "task_outcome": "Seven turns over 2 sessions and 1 handoff, all complete. I ran npm test: 39/39 pass. The final summary correctly covers both earlier and later work, and README, CLI and validation are present. Cost matched the recompute.",
   "notes": "The handoff itself was correct: read-before request sent, no tools, reply 3,696 chars with a blank-line junction and not cut off, no unseen heading. No stalls were recorded because the session had changed files, and there was no Progress check paragraph. The binding rule led, and <request> held that turn's words, with the seed listing the earlier requests and the open work. The failed Bash was the model's `set -e` demo hitting an expected non-zero exit. Cache ratio was 0.877, with reads in both sessions. No 'Learned in earlier sessions' section appeared, which is correct with 0 successes. D17-1/2/3 otherwise held: all 5 runs keyed to `npm test`, status unknown after a pipe, quoted `|` not split. Severity is low, and I am confident in the finding because it reproduced with the real code."
  }
 ]
}