{
 "model": "claude-haiku-5-5",
 "category": "context",
 "scenarios": [
  {
   "id": "c1-handoff",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Ran 4 sessions with 3 handoffs. Session 3 wrote ARCHITECTURE.md (167 lines) and session 4 finished it. It covers all 11 engine/ modules, the data flow and three risks. All 92 backticked identifiers in it exist in engine/*.ts. I checked: exit codes [0,0,0,0]; driver.err empty; leftovers empty; 57/57 tool-use/tool-result frames; all wire tool_use/tool_result pairs matched in 4 valid saved sessions; sessions 1-3 saved as interrupted with the handoff reason; 3 handoff .md files; the cost recompute $0.073380265 over 38 requests matches exactly, and every session has cacheReadTokens > 0.",
   "notes": "Read-before-handoff ran at all 3 handoffs: the 'Reading the latest results' status came first and the engine note was on the tool-result message. None of the replies ran a tool or was cut off (8,451, 4,817 and 4,617 chars). Each starts after a blank line ('...in parallel.\\n\\n## Handoff', '...trimmed.\\n\\nHandoff for the next session', '...outside `engine/`.\\n\\n## Handoff state'). Stalls are right: the frames carry 1, then 2, then none (session 3 wrote the file). Continuations 2 and 3 have 'Progress check: 1 session' and '2 sessions'; continuation 4 has none; none asks for sub-agents. Every continuation opens with the binding rule and quotes the original request in <request>. Old tool results were cleared twice (43,034→25,268 and 43,010→31,438). Session 2 re-read queryEngine.ts 215-459. Handoff 1 had told it to re-check those line numbers because its copy of the result was trimmed, although G-Code's TRIMMED_NOTE says trimmed results count as read. Session 3 likewise re-read 1-204 and 265-454 to confirm names. The seeds were accurate, so this is model behaviour. In session 4 the model sent one stray BrowserPlanCheck {plan_file:'none'}. It failed with a clear ENOENT and state 'failed', and the model admitted it was a mistake. The Skill tool is present among the 31 tools; no skill block, no MCP or DSH tools, and no DeferredToolCall."
  },
  {
   "id": "c2-compact",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "One turn: the context-full question was asked once, then the conversation was compacted 4 times and old results were cleared twice. Every request after a compaction succeeded, with no 400 or error frame. ARCHITECTURE.md (13.7 KB) has a section for each of the 11 modules, a data flow and three risks. I checked: exit 0, driver.err empty, leftovers empty, 89/89 frames paired, wire pairs matched, saved session valid, cost $0.04753697 over 19 requests matches, cacheRead 305,532.",
   "notes": "The summaries quote the user's request word for word and give the work state. One compaction began at 62,480 of the 60,000 budget: 11 parallel reads in one round pushed the count past the limit, and the check ran before the next request was sent. After compaction the kept results were clipped by design (at least 1,500 chars, each with a 'Truncated for context management' note). The model then read 'Tool output is capped at roughly 1,300–1,900 characters' into its summary and re-read in tiny sed chunks (57 Bash calls). That is a model misreading; G-Code behaved as designed."
  },
  {
   "id": "c3-stop-then-continue",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Turn 1 stopped with a clear error ('The context window is 93% full, so the turn stopped. Send /compact…'), and the turn was saved as interrupted with that reason. /compact took the conversation from 47,349 to 16,588 tokens and said the original was kept (compactArchives holds 1 archive of 6 messages). Turn 3 read only queryEngine.ts 515-967 and the end of hooks.ts, then wrote ARCHITECTURE.md with a section per module. No 400 errors. Exit 0, driver.err empty, leftovers empty, 17/17 frames paired, cost $0.015482555 over 10 requests matches.",
   "notes": "The saved compacted text has 3 part summaries ('Part 1 (of 3)', Part 2, Part 3 'transcript tail'). compact.ts:375 builds the per-part input as 'Part i of n' plus the <request>. Each part states the user's real request, not the interruption marker. Results are labelled with file paths, and every label keeps its file name. The hooks.ts block cut between parts is marked: Part 2 shows lines 1-186 and says 'the block continues in the next part', and Part 3 picks up at 187-274. Part 1 says its FileRead results fall in the next part, which is correct for a split between the calls and their results."
  },
  {
   "id": "c4-handoff-chain",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "4 sessions and 4 handoffs (the driver follows 3). SUMMARIES.md has 6 of 22 entries (engine/ compact, context, failureBudget, hooks, index, microcompact; 30 lines) with no duplicates. The task is unfinished, which this design allows. I checked: exit codes all 0, driver.err empty, leftovers empty, 37/37 frames paired, wire pairs matched in 4 valid sessions, cost $0.04173668 over 21 requests matches.",
   "notes": "G-Code did not cause the unfinished task. On a 40K budget Haiku read whole batches before writing anything (session 1 read 10 files and wrote nothing). It also re-read hooks.ts 4 times and queryEngine.ts in 500-line chunks; session 4 re-read 5 files without appending. Handoffs 1, 2 and 4 sent the read-before-handoff request. Those replies ran no tools, were not cut off (4,660, 5,674 and 4,597 chars) and start after a blank line. Handoff 3 skipped it correctly: two queryEngine.ts reads (30K and 26K chars) pushed the request to 48,963 of 40,000, so there was no room. That handoff's notes list those reads as 'arrived after the last reply, so they count as not yet read'. Stalls are right: 1 (nothing written), 0 (FileWrite), 0 (Bash append recorded in outputs), 1. Only continuation 2 has 'Progress check: 1 session'. All 3 continuations quote the ORIGINAL request in <request>, not an earlier handoff, and open with the binding rule. No loops."
  },
  {
   "id": "c5-huge-log",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "REPORT.md is correct. Top 5: W20 69,326, W10 46,455, E404 18,730, E500 11,674, E200 6,908. First E500 is 2026-10-25T00:00:15 and the last in file order is 20:26:37. I verified these with awk|sort|uniq and grep on the 12.6 MB, 160,000-line app.log. Exit 0, driver.err empty, leftovers empty, 6/6 paired, cost $0.00615204 over 7 requests matches, cache ratio 0.85.",
   "notes": "The model used grep, awk and sort in Bash, so outputs stayed small (the largest was 30,318 chars) and context stayed small. The FileRead paging path and the Bash head+tail path for huge outputs were not exercised. It correctly noticed the clock wraps at line 86,401 and reported both the file-order and the by-value E500 extremes."
  },
  {
   "id": "c6-many-files",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "INDEX.md lists all 150 notes exactly once, each under its right Topic (n mod 7); I parsed it and checked every note. Exit 0, driver.err empty, leftovers empty, 10/10 paired, no context-full, cost $0.01060471 over 7 requests matches.",
   "notes": "The model read 5 files, then all the notes with one Grep '.' over notes/ (18.5K chars), so the many-small-calls and compaction paths were not exercised. Its first self-check script reported false topic mismatches because of a bug in its own script. Its second check passed."
  },
  {
   "id": "c7-multi-turn-cache",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "All 4 turns completed. The explanation is right, JSDoc blocks were added (6), and count() was added with a test. The reported npm test result (6 tests, 1 pass, 5 failing seed tests) matches my own npm test run. Exit 0, driver.err empty, leftovers empty, 12/12 paired, cost $0.005598975 over 12 requests matches.",
   "notes": "Cache: 191,815 of 202,028 input tokens were cache reads (0.95), 10,181 were writes and 32 were uncached. The last request read 21,504 of 21,924. Per-request usage is not stored, but the totals mean every request read the prefix from cache, and the system prompt was frozen (one session). learned/patterns.json has key 'npm test' with runs 3, successes 0, broken 0. That is correct under D17-1: all three runs piped into tail or grep, so their status is unknown. The node --check run is not a verification. The model correctly left the failing seed tests alone and asked which side is right."
  },
  {
   "id": "j6-long-session",
   "status": "fail",
   "product_defects": [
    {
     "severity": "low",
     "title": "Learned-pattern parser splits shell commands on |, ;, && inside quotes: keys are rewritten and a check's own exit status is misread",
     "evidence": "runs-haiku-r17b/j6-long-session/home/.gcode/projects/56188d03…/learned/patterns.json. One key is `npm test | grep -E \"^(✖ | ℹ (tests | pass | fail))\" ; echo \"---\" ; npm test`, but the Bash command that produced it (log.jsonl, turn 5) was `npm test 2>&1 | grep -E \"^(✖|ℹ (tests|pass|fail))\"; echo \"---\"; npm test 2>&1 | tail -n 12`. A second key has `\"^(# (tests | pass | fail) | ✖ | ℹ (tests | pass | fail))\"` where the command had `\"^(# (tests|pass|fail)|✖|ℹ (tests|pass|fail))\"`. The quoted grep regex was changed: spaces were added around every |.",
     "location": "snapshot10w/src/memory/learnedPatterns.ts:72-79 (verificationOf: collapse(command).split(/\\s*(&&|\\|\\||;|\\|)\\s*/) ignores quoting; the key is rejoined with ` ${op} `)",
     "repro": "Splitting `npm test -- --test-name-pattern \"add|remove\"` with the same regex (I checked in node) gives [\"npm test -- --test-name-pattern \\\"add\", \"|\", \"remove\\\"\"]. The check's own exit status is then counted as 'unknown' and never as worked, and the key is cut to `npm test -- --test-name-pattern \"add` with an unbalanced quote. With a quoted | or ; before a check (e.g. `grep -q \"a|b\" f && npm test`), the status is 'exit', so the altered key can become trusted and be shown in a new conversation's system prompt. This contradicts LearnedPattern.key's own doc ('The command with its whitespace collapsed'). No effect on this run's task: the keys stayed untrusted (0 successes) and the session prompt was frozen. Confidence the code is wrong: high."
    }
   ],
   "task_outcome": "All 6 turns completed in one session with no handoff (53,747 of 90,000 used at the end). Validation and tests were added, then src/cli.js with tests and README.md. Turn 5 fixed the 4 failing seed tests, and turn 6's summary correctly covers all of today's work. My own npm test run: 35/35 pass. Exit 0, driver.err empty, leftovers empty, 26/26 paired, wire pairs matched, cost $0.027204865 over 19 requests matches, cache ratio 0.94.",
   "notes": "One FileEdit failed with a clear message ('Edit 6 of 6: String to replace not found… closest text is at line 103'), state 'failed', and the model recovered. The model changed seed behaviour in turn 5 (add accumulates, value rounds to cents, the lowStock rule) to make the existing tests pass, and it flagged the lowStock rule as a judgment call. Apart from the defect above, the learned-pattern statuses are correct: every run piped into tail or grep and is counted as runs only (successes 0)."
  }
 ]
}