LEARNINGS
Canonical source revision 2ccb1710c71144e239606b5a43536f70f2582714; source file last modified 2026-10-18T01:22:18.287Z; page rendered and canonical content read at 2026-10-18T01:22:20.067Z. Entry-level origin, disposition, verification, freshness, observation time, and follow-up are shown only when explicitly recorded in canonical metadata; missing values remain unavailable. This page does not read live fleet state.
# LEARNINGS — durable, provenance-tagged ground truth The knowledge pipeline (WS-8) starts here, now. Every line is one fact the fleet paid to discover, tagged with the run/PR that proved it. Rules: - **One fact per line. Each cites its source.** No fact without provenance. - These are *installed-version / empirical* truths — the kind a prompt is wrong about (Standing rule 7: distrust the prompt over the installed version). When one is contradicted by a later probe, append the correction with its own provenance; never silently edit history. - Promptsmith injects the matching entries into a rendered prompt as cited CONTEXT (W1-T19), so a worker inherits what we already learned.
01 / 5 entries
Canonical heading · line 14
- Claude Code's Bash-tool snapshot sources `$HOME/.zshrc` via `os.homedir()`; `ZDOTDIR` is IGNORED — worker shell isolation must set `CLAUDE_CODE_SHELL` (the rc filename follows the shell), not ZDOTDIR. [PR #8]
- The current `CLAUDE_CODE_SHELL=/bin/bash` isolation works only because `~/.bashrc` is ABSENT on this machine — an accident of this host, not construction; a populated `~/.bashrc` would silently isolate nothing. [PR #8 → W1-T17/T18]
- W1-T18 SHIPPED the general mechanism: every worker's `HOME` is redirected to a Remudero-controlled scratch dir (`config.workerHomeDir`, `<root>/worker-home`) holding ONLY empty rc files Remudero itself writes on every spawn, with `.claude`/`.config/gh`/`.gitconfig` symlinked back individually (never a wholesale HOME copy) — so isolation no longer depends on the operator's real `~/.bashrc` being absent, only on Remudero owning the path HOME points at. WS-0 FIELD FINDING 11c already established the Keychain OAuth token resolves off `USER`, independent of this redirection. Whether Claude Code ALSO needs `~/.claude/` state under the REAL (non-redirected) HOME to authenticate is UNVERIFIED LIVE — no subscription OAuth in CI (Standing rule 18/20) — so the `.claude` symlink-back is a defensive default, not an empirically-confirmed requirement yet; the live drill (planted `~/.bashrc` + still authenticates + `gh` still works) is W1-T12e. FALLBACK if the live drill finds the symlink-back insufficient: drop HOME-redirection for that one path and fall back to the W1-T17 preflight-probe-only posture (fail closed on contamination) while keeping the redirection for isolation itself. [W1-T18, this PR]
- Worker settings that fail schema validation are SILENTLY IGNORED under `claude -p` — a typo drops containment without erroring, so containment must be validated before spawn AND probed after. [WS-0]
- The SDK's `SandboxSettingsSchema` is `$loose` and silently strips unknown keys; validating against it alone PASSES a misplaced key (e.g. `allowedDomains` at the sandbox root) — validate shape explicitly, don't trust the schema to reject. [PR #6 / W1-T1]
02 / 2 entries
Canonical heading · line 22
- The Agent SDK yields the `type:"result"` envelope (`num_turns`, `total_cost_usd`, `subtype`) and THEN throws from the iterator on an error subtype — read the envelope before the catch, or a failed run looks free. [PR #8]
- `maxBudgetUsd` is checked BETWEEN turns: a $0.01 budget produced $0.21 of real spend. It is a circuit breaker with up to one turn of overshoot, NOT a hard cap — set budgets with headroom. [PR #8]
03 / 3 entries
Canonical heading · line 27
- Claude Code runs a BACKGROUND SELF-UPDATER that `npm install`s into the global prefix (`~/.npm-global`) mid-session; a worker spawn that lands in npm's unlink/relink window finds no binary and dies with ENOENT, which the SDK misreports as "native binary not found". Every live `claude` process has its own updater ⇒ a thundering herd of concurrent global installs under fleet concurrency. The same prefix also holds the `openclaw` CLI. Disable per worker (`DISABLE_AUTOUPDATER=1`, confirm empirically) and retry ENOENT-class spawns. [DIAGNOSIS.md, run W1-T1C-1784038021919]
- The SDK's "native binary not found" message fires iff `existsSync(exe) === false`; a bad/missing `cwd` yields a DIFFERENT message ("exists but failed to launch"). The message is DIAGNOSTIC — read which branch fired before theorizing. [DIAGNOSIS.md, run W1-T1C-1784038021919]- An `ls -l` taken AFTER a failure may show an already-REPAIRED state (the updater finished): timestamp your evidence with BIRTHTIME (`stat -f %SB`), not mere existence — the binary present post-mortem was born 8m after the crash, which is what named the race. [this session; the Architect first mistook the repaired state as "unchanged"]
04 / 4 entries
Canonical heading · line 33
- A doc that DESCRIBES a mechanism is never proof the mechanism EXISTS. PR #12 shipped `docs/review-gate.md`, passed CI, reported `verdict=merged`, and did NONE of its job (protection unchanged, no status ever posted). Acceptance proofs must be OBSERVABLE SYSTEM STATE — `gh api` output, a status object, a grep of the call site — never a file that talks about it. [PR #12/#13, this session]
- Splitting a task can ORPHAN its call site: T1C built the reviewer, T1D was to enforce it, and NEITHER owned `run-task.ts` CALLING it — the reviewer existed, fully tested, and was dead code for two PRs. When you split a task, name the integration point (the call site) as an explicit deliverable of one side. [PR #12/#13, this session]
- A required check must be posted by the ORCHESTRATOR, not solely by a worker: a crashed worker leaves the check ABSENT, and an absent required check blocks the merge forever (fail-closed becomes fail-stuck). The runner posts `remudero-review` unconditionally after judging; the fresh worker is advisory. [PR #13, this session]
- A required check with NO poster is a permanent DEADLOCK, so a manual/hand-run PR needs an escape hatch that invokes the SAME judge by hand (`rmd review <n>`) — never a `--force` bypass. Recovery if protection is ever misconfigured: `gh api --method PATCH repos/<owner>/<repo>/branches/main/protection/required_status_checks -F strict=false -f 'contexts[]=ci'`. [PR #13/this PR]
05 / 11 entries
Canonical heading · line 40
- A required-check context is the **JOB NAME**, not the workflow file/string — branch protection keys on the job, so renaming the job silently un-requires the check. [operator fleet: SynthWatch/neon-drift/OleyArcade]
- A **conditionally-skipped required check DEADLOCKS merge forever** (a `paths:`/`if:`-gated job that doesn't run reads as "expected, pending", never green; a skipped REUSABLE-workflow caller creates no check run at all under that nested context name, deadlocking it identically). The fix is ONE always-runs **CI-gate aggregator** job that make only the aggregator the required context — but a `needs:`-based aggregator only sees jobs WITHIN ITS OWN workflow file, so once required checks live in separate workflows (security/quality/architecture tiers), the aggregator must instead READ THE PR HEAD COMMIT'S CHECK-RUNS across all workflows via `gh api` (`checks: read`) and fail only on a real failure conclusion (W1-T24, corrected fleet pattern verified on craigoley/synthwatch). [operator fleet: SynthWatch/neon-drift/OleyArcade]
- **"0 violations" from a newly-added strict gate is SUSPICIOUS until FALSIFIED** with a planted probe that MUST fail (the neon-drift `_probe(x)` lesson) — an inactive gate and a clean gate report identically. [operator fleet: SynthWatch/neon-drift/OleyArcade]
- **Green tests that kill no mutants are test theater;** a mutation-testing baseline (Stryker for TS) is the falsifier coverage % cannot provide. [operator fleet: SynthWatch/neon-drift/OleyArcade]
- A **MAJOR dependency bump once caused a 28-minute production outage** ⇒ majors are EXCLUDED from auto-merge and carry a Dependabot `version-update:semver-major` ignore-rule; only minors/patches auto-merge behind the full gate. [operator fleet: SynthWatch/neon-drift/OleyArcade]
- **CodeQL default setup must be DISABLED when using an explicit CodeQL workflow** — the two conflict and both become unreliable. [operator fleet: SynthWatch/neon-drift/OleyArcade]
- **A diff-coverage run scoped to ONE test file OVER-REPORTS uncovered lines against a widely-exercised source file, so it is not a work list.** OBSERVED on `src/run-task.ts`: a scoped local run (`node --test` over the single touched test file, lcov to a scratch path) reported **20** flagged lines; CI's full-suite run over the same head reported **4**, and those 4 were a strict SUBSET of the 20. The 16-line gap is instrument artifact, not real coverage debt — `src/run-task.ts` is exercised by many suites, and a single-file lcov cannot see the coverage contributed by suites outside its scope. HYPOTHESIS, NOT ESTABLISHED: that the error is ASYMMETRIC — that a scoped run over-reports but never under-reports — which would make it SOUND as a pre-CI filter (nothing it calls clean can be dirty in CI) and UNSOUND as a work list. That asymmetry is assumed here, never measured: one observation cannot establish a direction. FALSIFIER, stated so it can be run: a single line that CI flags while a scoped run reads it clean kills the filter claim outright. Until then, use a scoped run only to CONFIRM lines CI already named, and take the flagged set from CI's own log. [commit e9dcb21, PR #2037] [subsystem: ci · files: scripts/diff-coverage.mjs, .github/workflows/ci.yml]
- **`cmd 2>&1 > file` CAPTURES NOTHING from a tool that reports on stderr, and the empty capture greps as all-clear.** OBSERVED: `2>&1` duplicates stderr to the CURRENT stdout (the terminal) BEFORE `>` redirects stdout to the file — so stderr goes to the terminal and the file receives only stdout. `diff-coverage` reports on STDERR, so that form captured an EMPTY file, and grepping it for flagged lines returned 0 for every line — reading as all-clear while the real output scrolled past on the terminal. The correct form is `cmd > out 2> err`, and any capture must be asserted NON-EMPTY (or checked against a control line known to be present) before a zero from it is believed. PROVENANCE for why this sits beside the entry above: this is what nearly mis-verified that finding — the zeros it produced were read as "every flagged line now covered" until a control line known to be present also read 0. [commit e9dcb21, PR #2037] [subsystem: ci · files: scripts/diff-coverage.mjs]
- **A HAND-OPENED PR BYPASSES `renderAcceptanceBlock`, SO IT MUST AUTHOR ITS OWN ACCEPTANCE BLOCK — AND THE RESULTING REVIEW FAILURE PRESENTS AS A DIFF PROBLEM WHILE EVERY CI CHECK IS GREEN.** OBSERVED on #2040 (zero `## Acceptance` headers in GitHub's stored body) and again on #2042: a PR opened via the REST API or `gh pr create` never passes through `renderAcceptanceBlock` (`src/lib/plan-pr-emitter.ts`), the emitter that guarantees a block `parseAcceptanceBlock` (`src/lib/review.ts`) can resolve — fleet-opened PRs get it automatically, hand-opened ones do not. A body with no `## Acceptance` header and no `Remudero-Task:` trailer gives `reviewCommand` (`src/run-task.ts`) nothing to resolve criteria from and it FAILS CLOSED, so the verdict reads as a failure on a PR whose checks are all green and the diff is the first place anyone looks. COROLLARY, and it is not optional: run `rmd check-acceptance` against GITHUB'S STORED BODY, never the local draft — on #2042 the two differed by a blank line GitHub had inserted, cosmetic in that instance but visible only to the check against the stored copy. [PR #2040, PR #2042] [subsystem: review · files: src/lib/plan-pr-emitter.ts, src/lib/review.ts, src/run-task.ts]
- **A POSTED VERDICT IS BOUND TO THE HEAD SHA, NOT TO THE PULL REQUEST, SO CORRECTING THE BODY ALONE CLEARS NOTHING.** OBSERVED: a failure verdict is never re-judged on the same sha, and repairing the body it was posted against leaves that verdict standing — clearing it requires a NEW HEAD, which an empty commit satisfies. The same sha-binding is what left #2037 reading `total_count: 0` on the combined-status endpoint once its head moved past the sha its review had been posted to: the verdict did not follow the PR forward, it stayed with the commit. [PR #2037, PR #2040, PR #2042] [subsystem: review · files: src/lib/review.ts, src/lib/sweep.ts]
- **GITHUB'S STORED PR BODY IS NOT NECESSARILY WHAT WAS SUBMITTED, AND AT LEAST TWO INDEPENDENT MECHANISMS PRODUCE THE DIVERGENCE — SO ANY CHECK ON A BODY MUST READ THE STORED COPY, NEVER THE LOCAL DRAFT.** OBSERVED twice, with different causes: on #2042 GitHub REFLOWED WHITESPACE, inserting a blank line inside the body it stored; on #2050 the SUBMITTING TOOL APPENDED 91 BYTES after the last authored line — a trailing separator and a generated-by footer — so the divergence arrived after submission rather than from the author or from GitHub's own formatting. Two independent causes, one effect, and neither is visible to a check run against the draft. On #2050 the insertion happened to land BELOW the Acceptance bullets, so `parseAcceptanceBlock` was unaffected and the block still resolved 3/3 — but that is POSITIONAL LUCK, NOT A GUARANTEE: the same 91 bytes landing between the header and the first bullet, or anywhere inside the bullet run, is one of the documented ways to reach `criteria parsed: 0` on a PR whose checks are all green. FURTHER EVIDENCE, AND A LEAD ITS OWN CONTROL ALREADY KILLED: the append was measured FOUR times on #2050 — 91 bytes on an MCP create, 0 on an MCP update, 58 on a `curl` PATCH update, and 0 on a second MCP update — so an update that appends and an update that does not, in the SAME PR, REFUTES the create-versus-update reading an earlier two-point version of this clause proposed. What co-varies across all four is the SUBMITTING PATH rather than the operation, and the footer TEXT differs between instances (one carries a session id, one does not), so these are not one mechanism observed twice. Recorded as an OPEN LEAD on four data points, never as established behaviour — the value of writing it down is that the next measurement can kill this reading too, exactly as the fourth killed the last one. [PR #2042, PR #2050] [subsystem: review · files: src/lib/review.ts]
06 / 4 entries
Canonical heading · line 54
- A lib function that EAGERLY calls `loadConfig()` (which shells `which claude`) FAILS IN CI, where the claude binary is absent. Resolve config LAZILY so injected-executor tests never touch it. The trap is STRUCTURAL — it applies to every future lib function, not just `containment.ts`. [PR #18, W1-T2]
- The reviewer's first live refusal (W1-T2 / PR #18) was CORRECT: the worker shipped an excellent HALF of the task (schema validation at the spawn boundary), wrote a confident PR body describing only that half, and passed CI 64/64. Static validation proves a settings file is WELL-FORMED; it never proves the sandbox ENGAGED — two criteria, two guarantees. Green checks are not evidence. [PR #18]
- When blocked by an acceptance gate, the one move that is NEVER available is editing the acceptance criteria to match the diff. Workers have write access to `plan/tasks.yaml`, so the prohibition must be STATED in every prompt, not assumed. [session doctrine]
- A gate that says only "no" costs a human round-trip on every block; a gate that NAMES the gap feeds straight back into the loop. The W1-T2 refusal said "1 criterion/criteria unmet" without saying WHICH — so `remudero-review` now names the first unmet criterion in the status description (full list in the ledger + PR comment). [this PR]
07 / 2 entries
Canonical heading · line 61
- A limit set NEAR the expected cost of the work is a WORK LIMIT, not a safety limit. Safety limits must sit an ORDER OF MAGNITUDE above the expected value so they only fire on pathology. This error was made TWICE — `maxTurns` (18 vs. ~36 needed) and `budget_usd` ($4 vs. ~$4 needed, W1-T3 killed mid-work). Set safety limits by asking "what number can ONLY mean something is broken?", never "what should this cost?". [W1-T3, PR #8]
- Calibration data (W1-T5 mounts.yaml) comes from OBSERVED runs, never from a guess in a chat. Observed so far: hello-world $0.41 · reviewer subsystem $2.26 · gate wiring $1.28 · containment probe ~$2.0 · W1-T3 was still working at $3.57 / 36 turns when its guessed $4 cap killed it. [W1-T3, PR #8]
08 / 3 entries
Canonical heading · line 66
- Tauri v2 targets 5 platforms from ONE codebase (Linux/macOS/Windows/Android/iOS; iOS renders in WKWebView), is independently security-audited (Radically Open Security), and has official App Store distribution guides; binaries are ~3–5 MB vs Electron ~150 MB. BUT its maintainers explicitly say **mobile is NOT a first-class citizen** — treat desktop as mature and mobile as young. [research 2026-07-14]
- Tauri's notification plugin is **LOCAL notifications**. Whether it supports **APNs REMOTE push is UNVERIFIED** — never assume remote push from a "notifications" plugin listing; a task must verify it before any design depends on it. (Remudero sidesteps this: push is an adapter concern — GitHub-mobile/ntfy/iMessage — not an app concern.) [research 2026-07-14]
- **Repo shape follows CONTRACT COUPLING**: everything that consumes the daemon API lives in ONE repo so a breaking change fails CI across all consumers atomically, in one PR; everything else (docs/site, commons, pro) lives OUTSIDE. Contract coupling decides repo shape — nothing else (D-5). [research 2026-07-14]
09 / 3 entries
Canonical heading · line 72
- A task whose criteria span multiple subsystems COLLIDES with one-concern-per-PR and is undeliverable. The worker's only honest moves are: do one concern and be blocked, or escalate. Decompose by CONCERN, never by convenience. Architect error, three occurrences: T1C (4 deliverables), T1D (orphaned call site), W1-T3 (4 subsystems). [PR #22, W1-T3]
- The reviewer judges DIFF + REPORT, never REPO STATE. A criterion satisfied by an EARLIER PR is permanently unsatisfiable by a later one. Its floor is keyword-matching the report against the proof text — crude and fail-closed by design, but it produces false positives on already-done work. (Fixes: `satisfied_by` is the manual patch; W1-T3F makes the reviewer OBSERVE repo state, the real fix.) [PR #22, W1-T3]
- Rule 15 held under pressure: blocked by 4 unmet criteria, the worker did NOT edit them. It scoped down, cited one-concern, disclosed the decision, and accepted the block. That is the behavior we want. [PR #22]
10 / 4 entries
Canonical heading · line 78
- Docs split by AUTOMATABILITY: changelogs, API refs, and interface specs are structured-in → structured-out and CAN be generated; conceptual guides, troubleshooting, and architecture decisions CANNOT — they need context absent from the codebase and must be written by a human. [research 2026-07-14]
- The governance that stops "shipping garbage at the speed of light" is to SEPARATE GENERATION FROM PUBLICATION: the machine generates and STAGES; the gate and the human APPROVE. (This is exactly what `rmd retro` does — deterministic gather + Architect synthesis staged as a gated plan PR.) [research 2026-07-14]
- Docs-as-code FAILS because it assumes humans will notice when documentation falls behind. The missing piece is the AWARENESS LAYER connecting a change to the docs it falsified — make it a reviewer-rubric obligation (W1-T30), not a hope. [research 2026-07-14]
- GitHub's repo-scoped agentic memory (Jan 2026) uses JUST-IN-TIME VERIFICATION with PRECISE CITATIONS TO CODE LOCATIONS, scoped to the repo, and lifted PR merge rates 83% → 90%. This validates provenance-tagged, repo-scoped learnings — and the READ side (inject them at prompt time) is exactly what we were missing (W1-T19). [research 2026-07-14]
11 / 6 entries
Canonical heading · line 85
- The core principle of context engineering: find the SMALLEST SET OF HIGH-SIGNAL TOKENS that maximize the likelihood of the desired outcome — context is a precious, FINITE resource, not a place to dump everything known. [research 2026-07-14, Anthropic "Effective context engineering"] [subsystem: knowledge · files: src/lib/learnings.ts, src/run-task.ts]
- Claude Code's OWN model is HYBRID: CLAUDE.md dropped in up front + glob/grep for JUST-IN-TIME retrieval, which "effectively bypasses the issues of stale indexing" — prefer retrieval (the plan is grep-able) over injecting whole files. [research 2026-07-14] [subsystem: knowledge · files: src/lib/learnings.ts, src/run-task.ts]
- Compaction's sharp failure mode: a summary that DROPS a critical fact is WORSE than no summary, because later steps trust it and cannot recover what was lost. Tune compaction for RECALL first, then precision; mitigate with ANCHORED SUMMARIZATION (goal/constraints/plan kept verbatim) and WRITE-BEFORE-COMPACT (externalize durable facts before the summary runs). [research 2026-07-14] [subsystem: context · files: src/lib/retro.ts, src/run-task.ts]
- A compacted message is REMOVED from the window and recoverable ONLY IF IT WAS STORED — durable findings must be written out (scratch file / ledger) as they are discovered, not held in-context. [research 2026-07-14] [subsystem: context · files: src/lib/retro.ts, src/run-task.ts]
- CACHE ECONOMICS: prompt caches key on EXACT PREFIX BYTES — any edit early in the context invalidates the cache for EVERYTHING after it; cache reads price at ~1/10th of fresh input. Assemble prompts STABLE-FIRST, VOLATILE-LAST. [research 2026-07-14] [subsystem: context · files: src/lib/learnings.ts, src/run-task.ts]
- Harnesses encode assumptions about what Claude cannot do on its own, and those assumptions GO STALE as models improve — question them frequently; a guard load-bearing at one model version can become dead weight at the next. [research 2026-07-14] [subsystem: retro · files: src/lib/retro.ts]
12 / 7 entries
Canonical heading · line 94
- The Agent SDK ships BUILT-IN tools (Read, Write, Edit, Bash, Glob, Grep, WebSearch, WebFetch, AskUserQuestion, Monitor, Agent/subagents); declaring them in `allowedTools` is enough, and `allowedTools` DOUBLES AS A SECURITY BOUNDARY — the agent cannot use a tool you don't list. [research 2026-07-14, Agent SDK docs] [subsystem: sdk-tools · files: src/lib/worker.ts, src/run-task.ts]
- `AskUserQuestion` is a BUILT-IN tool: Claude asks the user multiple-choice questions when uncertain — this IS the grill mechanism, natively. ★ UNVERIFIED whether it functions HEADLESSLY or only with a human at the terminal; the intake design branches on the answer — verify before wiring (W1-T42). [research 2026-07-14] [subsystem: feedback · files: src/run-task.ts]
- `WebSearch` is SERVER-SIDE (Anthropic's, the same engine Claude chat uses) ⇒ an Architect can research WITHOUT opening the sandbox's `allowedDomains`. ★ UNVERIFIED whether `WebFetch` is likewise server-side or a LOCAL fetch — that decides whether WebFetch touches containment (W1-T41). [research 2026-07-14] [subsystem: sdk-tools · files: src/lib/worker.ts, src/run-task.ts]
- SECURITY PRECEDENT: `claude-code-action` DISABLES WebSearch/WebFetch BY DEFAULT — fetched content is a prompt-injection surface. Grant them to the Architect only; implement workers never get them. [research 2026-07-14] [subsystem: sdk-tools · files: src/lib/worker.ts, settings/worker.json]
- ★ VERIFIED (live probe, re-run round 2 with an INDEPENDENT worker after round 1's proof was judged non-responsive — round 1 asserted a match without quoting the verifying worker's actual output or committing the artifacts, so it couldn't be checked): a worker CAN read a `plan/feedback/` image attachment via the Read tool's native multimodal support. Probe, end to end: (1) a 400×200 synthetic PNG was generated by hand (raw pixel buffer + `zlib.deflateSync`, no canvas/image lib) — navy background, a red rectangle upper-left, a green ellipse upper-right-of-center, and two horizontal yellow bars near the bottom (a long one and a shorter one below it); (2) the REAL `rmd feedback` CLI (`npx tsx src/run-task.ts feedback "the drain retry banner overlaps the status pill on narrow viewports" --attach shot.png --origin cli`) captured it, writing `plan/feedback/fb-1784433603527-48134e.yaml` and copying the image to `plan/feedback/attachments/fb-1784433603527-48134e/shot.png` — both committed in this PR as checkable evidence, not just referenced; (3) a SEPARATE, freshly-dispatched worker with zero prior context — given only the attachment's absolute path, no hint of its contents — opened it with Read and reported back verbatim: "The image has a dark navy blue background. Distinct shapes: a red rectangle in the upper-left area; a green (emerald) ellipse/oval in the upper-right-of-center area; and two horizontal yellow bars/stripes near the bottom of the image — a longer, thicker yellow bar spanning most of the width, and a shorter, thinner yellow bar directly below it aligned to the left side. No text-like marks are visible." — matching every shape, color, and position of the source image exactly, including correctly reporting no legible text (the two bars are plain color stripes, not rendered glyphs). MECHANISM: Read decodes and renders the image inline for the model — no OCR/vision API wiring needed on Remudero's side; `--attach` only needs to land the file on disk, which `resolveAttachments()` already does. Scope: this confirms PNG; other raster formats and non-image attachments (a "terminal dump" `.txt`/`.log` is already plain-text-readable) remain out of scope for this probe. [W1-T40 fix round 2, probe entry fb-1784433603527-48134e, independent-read agent a42ee1f6cb115de3d] [subsystem: feedback · files: src/lib/feedback.ts, plan/feedback/fb-1784433603527-48134e.yaml, plan/feedback/attachments/fb-1784433603527-48134e/shot.png]
- ★ VERIFIED (W1-T42, ★ CORRECTS the line above — never silently edited, the ★ UNVERIFIED line stays for history): `AskUserQuestion` does NOT function headlessly and it does NOT stall/hang either — it silently AUTO-RESOLVES with an EMPTY answer in ~37ms when there is no TTY, because Claude Code runs as a subprocess with no terminal attached and nowhere to render the question's UI, so the tool call just quietly "succeeds" with nothing collected (documented independently in `anthropics/claude-code` issues #50728 and #30983, and `anthropics/claude-agent-sdk-python` issue #327 — a `canUseTool`/`PreToolUse` handler does not help; the SDK resolves the tool before either is awaited). This is WORSE than a hang for `rmd triage`: a hang burns turns visibly, but an empty auto-resolved answer looks like a normal tool result a worker could silently trust and act on. Load-bearing structural fact: `rmd triage`'s Architect ALWAYS runs via `spawnWorker` (`src/lib/worker.ts`'s Agent SDK `query()`), i.e. always a subprocess with no TTY, REGARDLESS of whether the shell that typed `rmd triage` has one — so "AskUserQuestion at the terminal" (the interactive branch MASTER-PLAN §7B names) is not a live option this architecture can reach; the async branch (a `needs-human` GitHub issue reusing `lib/escalate.ts`, MASTER-PLAN §4) is the ONLY viable grill mechanism, unconditionally, not merely a no-TTY fallback. Wired: `TRIAGE_WORKER_TOOLS` never lists `AskUserQuestion` (confirmed correct, previously justified only by the general "no live operator" learning below); `.remudero/skills/feedback.yaml`'s `tools:` no longer lists it either. [research 2026-07-19, github.com/anthropics/claude-code#50728, #30983, github.com/anthropics/claude-agent-sdk-python#327] [subsystem: feedback · files: src/run-task.ts, src/lib/triage.ts, src/lib/escalate.ts, .remudero/skills/feedback.yaml]
- ★ VERIFIED LIVE (W1-T42, ★ CONFIRMS the two lines above from inside a real headless worker — the missing first-hand probe the review demanded before the grill is built on it): ASSERT — I invoked `AskUserQuestion` from within THIS headless fix-worker (the same no-TTY `spawnWorker`/Agent-SDK `query()` subprocess context the triage Architect runs in) and the observed behavior was that the tool IS NOT PRESENT IN THE WORKER'S TOOL MANIFEST AT ALL: `ToolSearch "select:AskUserQuestion"` returned VERBATIM `No matching deferred tools found`, and a keyword `ToolSearch "ask user question multiple choice prompt"` surfaced only unrelated tools (TaskStop, Monitor, WebSearch, WebFetch, ExitWorktree, EnterWorktree, SendMessage, file_upload, Gmail-auth) with NO `AskUserQuestion` among them — there is no schema to fetch, so the tool cannot even be called. So the worker-side failure mode is not the ~37ms empty auto-resolve documented above (that is what a session which HAS the tool but no TTY does); it is one rung earlier — a triage worker never RECEIVES `AskUserQuestion`, because `allowedTools` (`TRIAGE_WORKER_TOOLS`) doubles as the security boundary and never grants it (the same "the agent cannot use a tool you don't list" invariant recorded above). Therefore the async grill MUST route ambiguous questions through the `needs-human` ledger / `escalate()` path (`buildGrillEscalation` → GRILL-class Escalation → GitHub issue, `src/lib/triage.ts`), NOT a live `AskUserQuestion`, because the tool is unreachable to the worker by construction — even if it were granted, the no-TTY auto-empty-resolve above would silently corrupt the answer. Absence-as-finding is an ACCEPTABLE and SUFFICIENT verification here: the empirical live probe (tool not in manifest) and the research (tool auto-empties without a TTY) agree, and both point to the same single viable mechanism. [W1-T42 fix, PR #292, live probe run-W1-T42-1784515310302] [subsystem: feedback · files: src/run-task.ts, src/lib/triage.ts, src/lib/escalate.ts, .remudero/skills/feedback.yaml]
13 / 4 entries
Canonical heading · line 104
- Playwright is cross-browser (Chromium/Firefox/WebKit), headless-native, CI-native, with both MCP and CLI for agents. Its MCP uses ACCESSIBILITY-TREE snapshots (2-5KB structured YAML) not screenshots (500KB-2MB images) — 10-100x cheaper in tokens for structural work. The CLI saves compact YAML snapshots (~4x fewer tokens) but REQUIRES SHELL ACCESS; MCP is the sandboxed-client path. [research 2026-07-14] [subsystem: design-review · files: settings/worker.json, src/lib/containment.ts]
- ★ SECURITY: Playwright's `browser_run_code_unsafe` runs ARBITRARY JS in the Playwright server — it is RCE-EQUIVALENT and "only for trusted MCP clients." A bypass-mode worker is not that: HARD-DENY it in every worker profile. [research 2026-07-14] [subsystem: design-review · files: settings/worker.json]
- Playwright needs a real browser with NETWORK EGRESS — incompatible with the locked worker `allowedDomains` UNLESS a dedicated Design-Review profile EXPLICITLY allowlists the browser/target domains. Browser egress is a per-skill grant, never global. [research 2026-07-14] [subsystem: design-review · files: settings/worker.json, src/lib/containment.ts]
- Screenshots blow up token usage (500KB-2MB each) — request them "only when necessary," for artifacts a human must eyeball; use accessibility-tree snapshots for everything structural. [research 2026-07-14] [subsystem: design-review · files: src/lib/containment.ts]
14 / 1 entry
Canonical heading · line 111
- Docs are a GATED ARTIFACT, split by automatability: GENERATED docs (rmd --help, CLI reference, API ref, CHANGELOG) drift from code unless CI enforces byte-equality (regenerate + diff = RED) — the anti-rot mechanism; AUTHORED docs (concept/architecture/troubleshooting) are reviewer-gated (the docs rubric). ONE command registry feeds rmd --help + the GitHub /docs + the website — never three hand-copies. [research 2026-07-14] [subsystem: docs · files: src/run-task.ts, docs]
15 / 3 entries
Canonical heading · line 115
- An acceptance proof only EXECUTES if it carries the exact house dialect `unit test: <file-or-test-name>` (or `grep: …`) — the text after the prefix is a file or test NAME passed to a name-filter, NOT prose. W1-T181 shipped with criteria 3/5 carrying no prefix and the prefixed ones carrying prose, so every proof read not_executable or executed_fail via an empty name-filter; the floor was working and the PROOFS were unrunnable. Operator authoring error, not a floor defect. `rmd review` from the main checkout ALSO cannot execute at all (no PR-head materialization), so an operator-posted verdict is keyword-only by construction. [PR #411, W1-T181 → W1-T185]
- The reviewer's keyword floor was CASING/SEPARATOR-BLIND: `maxTurns` (criterion) vs `max_turns` (code/report YAML) tokenized as a MISS and false-blocked a correct PR (#42, W1-T5). Normalized in review.ts (split camelCase before lowercasing so maxTurns ≡ max_turns ≡ max-turns); the real fix is W1-T3F (observe repo state, don't match report keywords). [PR #42, W1-T5]
- A criterion can go STALE when a LATER PR relocates a concept: PR #23 moved budget OFF the mount to per-task `budget_usd`, but W1-T5's criterion still named `maxBudgetUsd` as a mount knob. The Architect corrects a stale criterion via a plan PR; a worker never. Honesty test: nothing dropped or weakened — only realigned to the shipped architecture (Rule 15/16). [PR #42, W1-T5]
16 / 4 entries
Canonical heading · line 122
- AN OBSERVED GAP NAMED ONLY IN PROSE IS A LEAK — file it as a task or a criterion AT OBSERVATION TIME, every time. While fixing the worker commit contract (PR #407) the agent's own survey recorded that `renderFixPrompt` carried no commit-message guidance, wrote that finding into a report, and did not file it. Hours later a fix-rung worker authored a 111-character header and blocked PR #423, producing escalations #427/#428 and a rebuild — the exact gap, already seen and written down. Reading #407 back also exposed two further defects it had shipped (the contract block inserted mid-sentence, and guidance that a leading acronym 'is fine' which an adjacent passing test disproved). A report is not a backlog: prose decays, a filed criterion does not. [PR #407 -> #427/#428 -> PR #430]
- WRONG-FAST MASKS SLOW-RIGHT: repairing a data source REQUIRES re-measuring its consumers, because a broken source is often *fast* at being wrong and the repair reveals a cost that was always there. W1-T181 fixed the batched gateway's ENOBUFS (it had been throwing and returning `[]` in milliseconds, so the board was WRONG but quick); with real data flowing, the same console went from ~23s to ~49s first paint, exposing W1-T187's O(tasks x ledger) projection defect that the empty result had been hiding. A fix that improves correctness and appears to "cause" a slowdown has usually uncovered one. Re-measure the consumers, do not attribute the regression to the repair. [W1-T181 -> W1-T187, PRs #411/#425]
- INVERTED PROTECTION — audit UNATTENDED paths for guards that exist ONLY on operator-watched ones. The interactive verb is where a human is already looking; the unattended rung is where nobody is. `routeFix` (the `rmd fix` operator verb) opened with `if (prState !== "OPEN") return refused`, and that guard was reachable ONLY from `fixCommand` — the sweep-driven rung reached `runFixRung` directly and had no equivalent, so it spent strikes, dollars and escalation issues on already-merged PRs (~$2.54 and 76 worker turns in one evening). Its own doc comment advertised the refusal as though it were a property of the rung. Whenever a safety check is found on a hand-driven path, ask which automated path skips it. [W1-T177, PRs #395/#401/#417]
- READ FRESH STATE IMMEDIATELY BEFORE INTERVENING — the machine may already be mid-remedy, and a diagnosis is stale the moment it is written. Three times in one session an assessment was overtaken while being written: PR #411 was judged at commit `f743685` (1 of 5 criteria) while the fix rung landed `90841b7` completing the work; PR #388 was called genuinely-red while a rung landed `1fbea36` fixing it; a CI failure was mid-investigation while `b294343` already fixed it. Re-read PR/ledger state at the moment of acting, and when reporting an assessment, state the commit it was made against. [PRs #411/#388, commits 90841b7/1fbea36/b294343]
17 / 1 entry
Canonical heading · line 130
- The SDK `type:"result"` envelope carries the telemetry a ledger task must log — verified from the installed `sdk.d.ts` (SDK 0.3.209): `num_turns`, `total_cost_usd`, `usage` (NonNullableUsage aggregate), and `modelUsage: Record<MODEL_NAME, ModelUsage>` (the MODEL(S) used are the map KEYS; each `ModelUsage` has `inputTokens`, `outputTokens`, `cacheReadInputTokens`, `cacheCreationInputTokens`, `costUSD`, `contextWindow`). **EFFORT is NOT in the envelope** — it is an INPUT (`Options.effort`/the mount), so log the configured effort. W1-T6 spent ~33 of 61 turns re-discovering this; it is now an injectable learning tagged to worker.ts/ledger.ts/run-task.ts, and W1-T6 carries a `files:` field so the matcher selects it (before, repo-wide, it lost the budget tie and was DROPPED). [PR #42/W1-T6, DIAGNOSIS.md]
18 / 2 entries
Canonical heading · line 134
- ★ A criterion requiring LIVE OPERATOR confirmation ("operator confirms", "user selects") is STRUCTURALLY UNFIT for a headless worker — there is no TTY and no operator. W1-T9 burned its last ~15 turns writing readline-reproduction scripts (`rl-repro.mjs`) trying to conjure an operator that isn't there, then died `error_max_turns` — it would overrun at ANY budget. Redesign interactive steps for no-TTY: expose `--tier`/`--yes`-style FLAGS + a TTY-ABSENT SAFE DEFAULT (log the chosen value), prompt ONLY when `process.stdin.isTTY`, and test the non-interactive path via injected input/flags — never a live human. A new error class, distinct from over-scoping and looping. [DIAGNOSIS.md diag/w1t9-max-turns, W1-T9] [subsystem: cli · files: src/lib/init.ts, src/run-task.ts, bin/rmd]- `~/.claude.json` exposes tier-relevant KEYS locally with no network call — `hasAvailableSubscription`, `cachedUsageUtilization`, `modelAccessCache`, `oauthAccount` (tier discovery rung 2); the `/usage` output (rung 1) is parsed by `headroom.ts` (W1-T4) and is machine-readable headless. Feed BOTH as pure INPUTS to `detectTier` so detection is unit-testable over fixtures (max5x/max20x/ambiguous) with no live calls — the weak-residue learning the diagnosis flagged, so tier detection isn't first-time-discovered again. [DIAGNOSIS.md diag/w1t9-max-turns, MASTER-PLAN §9(h)] [subsystem: platform · files: src/lib/tier.ts, src/lib/init.ts, src/lib/headroom.ts]
19 / 2 entries
Canonical heading · line 139
- The daemon task (W1-T12) bundled THREE concerns (scheduler loop + launchd unit + crash-recovery) AND three HEADLESS-UNFIT criteria — a live OVERNIGHT drain, a `launchctl`-LOAD into a real session, and a live KILL-and-recover — then died `error_max_turns` at 81 turns / $10.27 (the 4th such event). A headless worker cannot verify launchd/boot/kill behavior; those are `verify:human` COMMISSIONING steps. Split the daemon into headless unit-tested LOGIC (scheduler loop over an injected clock+runner; plist GENERATION + in-process boot assertion; recovery over a SEEDED interrupted state) and a separate live-commissioning task — the machine builds + unit-proves, the human commissions live (draft→playtest→ready). [DIAGNOSIS.md diag/w1t12-max-turns, W1-T12] [subsystem: architecture · files: src/lib/daemon.ts, src/lib/launchd.ts, src/run-task.ts]
- Standing rules 18/19 were enforced FORWARD-ONLY: W1-T12 pre-existed them and violated both, yet slipped through to a worker because nothing re-graded the already-authored queue. A new sizing/fitness rule must RETROACTIVELY sweep every open task; `rmd retro` re-grades the queue against every standing rule and files corrective tasks (Standing rule 20). A rule the queue is never swept against protects only tasks written after it. [DIAGNOSIS.md diag/w1t12-max-turns, W1-T12] [subsystem: process · files: plan/tasks.yaml]
20 / 2 entries
Canonical heading · line 144
- The runner was the FIRST line of task-quality defense, so FOUR malformed tasks (W1-T6, W1-T9, W1-T12) reached workers and burned budget before the pattern was caught. Task quality must be gated BEFORE dispatch — a deterministic linter, FAIL-CLOSED (`verdict=blocked_illformed`, no worker spawned) — and re-graded continuously by the retro plan-health sweep. Reactive diagnosis after an 81-turn / $10 burn is the anti-pattern this project exists to kill; the runner is the LAST line, not the first. [W1-T12, this session] [subsystem: architecture · files: src/lib/task-linter.ts, src/run-task.ts, src/lib/retro.ts]
- Most task-quality failures are catchable DETERMINISTICALLY — no LLM: criteria count (sizing, Rule 19), subsystem count, forbidden live-context verbs (headless-fitness lexicon, Rule 18), proof-shape (observable proof, not a vibe). Reserve the Architect LLM review for genuine ambiguity ("is this really one concern?", "is a criterion secretly a task-inside-a-task?"). Cheap deterministic checks first; the LLM layer is advisory and does not block. [W1-T12, this session] [subsystem: architecture · files: src/lib/task-linter.ts]21 / 1 entry
Canonical heading · line 149
- An unknown CLI command / unrecognized subcommand fell THROUGH to launching a drain — `rmd daemon install --dry-run` silently ran the daemon and merged a task unattended (bogus `install`/`--dry-run` were ignored) — and STOP was a persistent latch that silently blocked every future `rmd drain` until manually cleared. Both are unattended-daemon HAZARDS. Fix: a control surface must FAIL LOUD on bad input (unknown command/arg → usage + non-zero exit + spawn NOTHING; spawning commands validate flags BEFORE touching config/locks/workers), and STOP must be ONE-SHOT (auto-consumed on the halting run's terminal verdict; a no-op-that-warns when nothing is running), genuinely distinct in lifecycle from PAUSE (a persistent maintenance hold cleared only by `rmd resume`). [fix/cli-safe-control-surface, W1-T14/T15 incident] [subsystem: cli · files: src/run-task.ts, src/lib/fleet-control.ts]
22 / 1 entry
Canonical heading · line 153
- `rmd daemon` originally read its plan from the CHECKOUT it ran in and hardcoded the status gateway to `ghGateway(owner, "remudero")` — so it could only drain the main repo and an unattended run would SILENTLY self-target. It now takes an EXPLICIT target: `--repo <name>` scopes the gateway to `<owner>/<name>` and reads the plan from a synced clone (`<root>/repos/<name>/plan/tasks.yaml`, `reset --hard origin/main` so it's current); `--plan <path>` overrides; `--dry-run` previews (spawns nothing, logs `daemon.target`). A SELF-TARGET GUARD refuses to drain the daemon's own source repo unattended without `--allow-self-target`. `rmd daemon-plist --repo <name>` bakes `--repo` into the launchd unit's ProgramArguments so the unit drains the intended repo, not an implicit default. (Task EXECUTION already derived its repo from `task.repo` via runTask — only the daemon's plan source + gateway were pinned.) [fix/daemon-repo-targeting, W1-T12d] [subsystem: cli · files: src/run-task.ts, src/lib/launchd.ts]
23 / 1 entry
Canonical heading · line 157
- The launchd unit runs the daemon with **KeepAlive**, so launchd restarts it on **any** exit. This IS the crash-recovery path — a kill-9 → KeepAlive restart → the daemon re-derives merged state from GitHub and goes idle (reconstructOrphan, W1-T12c, handles a real in-flight orphan), with **no duplicate task run**. **Gotcha:** a `--max N`-bounded daemon under KeepAlive **restart-loops** — it drains N, exits `max_reached`, launchd restarts it, it finds nothing runnable (`daemon.idle`), exits, restarts… (observed in commissioning: 3 `daemon.start` lines, the 2nd/3rd immediately idle). For a real unattended run use **no `--max`** (persistent poll loop); reserve `--max`/`--dry-run` for a bounded manual drill and expect KeepAlive to bring it back. Plist: `~/Library/LaunchAgents/com.remudero.daemon.plist`; commission with `launchctl load <path>`. [chore/t12d-commissioning-record, W1-T12d] [subsystem: platform · files: src/lib/launchd.ts, src/lib/daemon.ts]