{
 "model": "claude-haiku-5-5",
 "category": "build",
 "scenarios": [
  {
   "id": "a1-static-site",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Built the 5-page Panaderia Sol site with css/styles.css, js/main.js, js/validate.js as an ES module, data/menu.json and 7 SVGs. I reran npm test (`node --test`): 22/22 pass. All 67 local href/src links resolve, and main.js has the Escape/Arrow lightbox and the fetch of data/menu.json. The agent's curl of every page and menu.json returned 200, and TaskStop stopped the server (exit 143, final tasks frame 'stopped').",
   "notes": "36 tool calls in one turn: 33 tool-use frames plus 3 refused BrowserPlan calls, which have a tool-result frame and no tool-use frame, as designed. Those 3 ('Not run: invalid input', unknown key selector) saved as state 'refused'; the model corrected them and the next 3 completed. 3 BrowserVerify runs passed (13/11/9 checks). 39 outputs, all with baseline absent:true, and each last version's sha matches disk. Only the engine's .gcode/verification/.gitignore is not listed. 36/36 tool_use paired, exit 0, empty .err, 1 turn-complete, leftovers empty. The agent did not curl the port after stopping, but the runner's leftovers are empty and my lsof of 8700 is clean. Cost recompute matches ($0.034086245); cache reads 599,842 over 15 requests. 31 tools with TeamMemoryWrite and the Skill catalog (6 skills), no mcp__ tools, no auth/dsh folder or mcp.json in any run HOME. Heartbeat at 20s and 35s, then it stopped when output arrived. learned/patterns.json: 'node --test' has runs 1, successes 0, correct because of `| tail`."
  },
  {
   "id": "a2-vite-react",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Scaffolded Vite react-ts non-interactively, installed, and started the dev server in the background on 8705 with --strictPort. Curl confirmed the page. The h1 was changed to 'Hello from G-Code' and the served /src/App.tsx contains it. The server was left running as asked. I ran `npx tsc -b --noEmit` (exit 0) and a vite build into a temporary folder (exit 0, then removed).",
   "notes": "All 3 tasks frames show the server running at turn end, and TaskOutput says running. result.json leftovers are empty, but the runner checks 5 s after exit, so the ~3 s bound in the check text was not measured directly. The port and processes are free now. 10 tool calls, all paired; exit 0, empty .err. 19 outputs match disk. Cost recompute matches ($0.003361085); cache reads 165,256. No verification command ran, so no learned file, which is correct. npm is on PATH. The model wrote /tmp/index.html (model choice)."
  },
  {
   "id": "a3-vite-build-preview",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Scaffolded vanilla-ts, installed, and added an origins section (Ethiopia, Colombia, Guatemala) to src/main.ts. I reran `tsc --noEmit` and a vite build into a temporary folder: both clean. dist has index.html and the JS asset, and the bundle contains the origins. The agent curled index.html and the JS asset (200), stopped the server with TaskStop (exit 143), then confirmed by curl that the port was closed.",
   "notes": "Baseline check: the scaffold was created inside this turn, so src/main.ts and src/style.css correctly have baseline absent:true. versions[0] of main.ts holds the original scaffold (setupCounter present, no origins) and versions[1] the edit. The scenario's wording about the baseline is stale; the original content is kept as the first version. dist is not listed in outputs, while every other non-node_modules file on disk is. 13 tool calls, all paired. Cost recompute matches ($0.00483941). Learned 'npm run build' has runs 1, successes 0, correct because of `; echo ...`."
  },
  {
   "id": "a4-canvas-game",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Built Snake: index.html, game.js (ES module) and logic.js with an injectable RNG. I reran npm test: 66/66 pass. The test-edit-retest loop completed without any failure-guard stop. BrowserVerify passed (23 checks, 13 assertions).",
   "notes": "One FileEdit had invalid JSON arguments. It saved as state 'refused' ('Not run: ... not valid JSON', nothing executed), the model redid it next reply, and the saved session pairs it (19/19). It has no tool-result frame in the driver log, which matches attempt b's j1 note. Heartbeats: 'Still waiting' at 20s, 'Still thinking' from 35s to 1m20s, and another at 20s in a later reply; each run stopped when output arrived. Cost recompute matches ($0.04935437); cache reads 1,106,017 over 20 requests. Learned 'npm test' runs 3, successes 0 is correct (all piped). Observation, not counted as a defect (fairly sure): firstSeen is the time of the last run in the first saved batch, not the first run. Here firstSeen is 1793832062188 while the first `npm test` frame is at 1793831997903, because notePatternOutcome overwrites `at` and saveLearnedPatterns uses it for firstSeen (src/memory/learnedPatterns.ts). A grep of snapshot10x/src finds no reader of firstSeen outside this file, and this file only validates and writes it, so no behaviour is affected. It is worth a one-line fix when convenient."
  },
  {
   "id": "a5-md-docs",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Wrote zero-dependency build.mjs (markdown subset, shared layout, nav, TOC) and test/build.test.mjs. I reran npm test (38/38) and the build (exit 0): site/ has all 6 pages and the rebuilt files are byte-identical to what the agent produced. BrowserVerify passed 7 checks.",
   "notes": "Changes is complete: 15 outputs cover everything on disk except the 6 unchanged seed files and the engine's .gcode/verification/.gitignore. One BrowserPlan was refused for invalid input and saved as 'refused'. One Bash ran and exited 1 (`cat package.json` was last) and stays 'failed', with the exit code in its text. Known harness issue: seed backticks were eaten by printf, so the content has no fenced code and the second paragraph lacks its topic word. Not G-Code. scenarios.json says effort medium, while result.json says high, as the brief states. Heartbeats ran 20s to 1m50s and then stopped. Cost recompute matches ($0.03495039). Learned 'npm test' runs 2 and 'npm run build' runs 1, both successes 0, correct because of pipes and `; echo`; the quoted `|` in the grep did not split the command."
  },
  {
   "id": "a6-express-api",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "npm init plus express 5.3.0; in-memory /api/notes CRUD and public/index.html using fetch. The agent exercised every endpoint in the background on 8725, stopped it with TaskStop (exit 143) and confirmed the port closed. I started it myself on 18725: POST 201, GET 200, PUT 200, DELETE 204, missing note 404, GET / 200. I then stopped it and the port is free.",
   "notes": "9 tool calls, all paired; exit 0, empty .err, leftovers empty. Stopping `npm start` also ended its node child (the agent's curl after the stop got connection refused). 4 outputs match disk. Cost recompute matches ($0.005313535). No check command, so no learned file. The model said plainly that it did not load the page in a browser."
  },
  {
   "id": "a7-chinese-site",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Four Chinese pages share styles.css, and menu.js fetches menu.json and shows CNY prices. I reran check.mjs (exit 0, all 4 pages have lang=zh-CN and a title). Project files, log and saved session have no U+FFFD or mojibake markers, and the final summary is in Chinese.",
   "notes": "One BrowserVerify ran and failed on the model's non-unique selector .menu-item .dish-price (strict mode, 13 elements). It kept state 'failed', with a single 'Error:' prefix and report/failure.png/trace paths; its report.json says failed. The corrected plan passed (10 checks, 8 assertions). The model deleted its first plan file with Bash; Changes records that with deleted:true, its versions kept and an absent baseline. One English progress sentence in the reply is model text. The message has no 浏览器…验证 trigger and no skill block appears, which is correct. 20/20 paired. Cost recompute matches ($0.01643154). `node --check` was correctly not learned, so no learned file."
  },
  {
   "id": "j1-todo-fullstack",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Built the Node http server with a JSON file store, REST API and vanilla JS frontend (add/toggle/delete/filter). I reran npm test: 59/59 pass, using ephemeral ports, with the file list unchanged afterwards. The agent started the app in the background on 8955, curled the main endpoints, stopped it with TaskStop (exit 0 through its SIGTERM handler) and confirmed the port closed.",
   "notes": "The one failed Bash was the model's probe (`cat package.json` with no file, exit 1), correctly 'failed'. Tests passed on the first run, so no retest loop was needed. 20/20 paired; 11 outputs match disk. Heartbeat at 20s and 35s, then it stopped. Cost recompute matches ($0.027007655). Learned 'npm test' runs 1, successes 0, correct because of `| tail`. The model removed its smoke-test data folder itself."
  },
  {
   "id": "j2-cli-tool",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Wrote csvstat.mjs (quoted fields, commas inside quotes), gen-sample.mjs and tests. I reran npm test: 32/32 pass. Python's csv module reads 2,000 rows in sample.csv. I ran the tool on it: exit 0, with per-column count, distinct and numeric min/max.",
   "notes": "The first run used a bad `node --test test/` script (model error: 1 failing pseudo-test), fixed in package.json. 31 tests passed next, and a later edit added one, giving 32. The earlier 'All 31' text is accurate for its moment. 15 tool calls, all paired. Cost recompute matches ($0.01873175). learned/patterns.json matches the commands: 'npm test' runs 2 and 'npm run gen 2>&1 && wc -l sample.csv && head -5 sample.csv && echo ---- && npm test' runs 1, all successes 0 because of `| tail`/`; echo`. Observation, not counted as a defect: only the last check part has its redirections removed, so `2>&1` stays in that key's prefix. The effect is cosmetic: at worst two textual variants become separate patterns, and this one is not trusted or shown."
  },
  {
   "id": "j7-sonnet-sanity",
   "status": "pass",
   "product_defects": [],
   "task_outcome": "Wrote TABLE.md with the 8 planets (diameter, moons) and verify-table.js. I reran the script: 8 rows, PASS, exit 0.",
   "notes": "Small baseline run: 2 requests, 3 tool calls all paired, 14 s, cache reads 24,395. Cost recompute matches ($0.00141305), and all 58 scenarios recompute with max difference 0 ($2.149071 total). The model id in the run is claude-haiku-5-5; the scenario name is historical. No heartbeat, no learned file (a plain node script is not a check), no GPCI text in the log."
  }
 ]
}