T
Sign in

Correction line — incident report & runbook (vps-d3go)

All docs GitHub source

Correction line — incident report & runbook (vps-d3go)

Date: 2026-09-30 Host: vps-d3go (2 vCPU, 6 GiB RAM, 512 MiB swap) Services: queue-watch.service, opencode.service, project@trading.service, caddy, cloudflared

This is the reference for the next time the line misbehaves. It records the symptoms, the root causes, everything that was changed, the architecture fix, and how to diagnose and recover in minutes.

0. Current state (end of 2026-09-30)

  • Suite is green: 3261 passed / 0 failed (251 files).
  • Parallel across independent areas, serial within one: the planner emits one workstream per file-disjoint feature area; up to 3 run at once (SWARM_MAX=3), while rows that share files stay in a single lane. So it is safe: workstreams never overlap, and a lane is reset to master at start. SWARM_SHIP_AREAS=2, SWARM_SHIP_EVERY=600 (batch a couple of ready lanes into one suite+deploy).
  • Caps: MemoryMax=3G, CPUQuota=200%. Runtime RAM ~1.3 GB (was 5 GB+), load ~7 (was 84). No OOM.
  • All swarm/ws* branches are at master (nothing stranded).
  • Doctor reports healthy. 10-minute cron tripwire installed.
  • Master HEAD at the end of the work: 28fe439.
  • Two copies of this doc: docs/INCIDENT-2026-09-30-resource-spikes-and-rewrite-loop.md and /root/INCIDENT-2026-09-30-queue-loop.md.

1. The two symptoms

  1. Resource spikes: RAM hit 5+ GiB, swap engaged, CPU 96%+, load up to 84 on 2 vCPU (measured 16:10–16:50; the box rebooted at 17:29).
  2. Duplicate work: workers re-implemented correction rows that were already finished — the same row built again and again.

They are one incident: the correction-line swarm was both the resource consumer and the thing stuck in a re-work loop.

2. Architecture (the moving parts)

queue-watch.service
  └─ scripts/queue-watch.sh run            # arms via .queue-auto; peeks the board
       ├─ scripts/queue-doctor.sh          # circuit breaker (also its own 10-min cron)
       ├─ scripts/unblock.sh once          # stuck-row supervisor (QUEUE_UNBLOCK=1)
       └─ screenshots/.swarm-run.sh run    # snapshot of scripts/swarm.sh (ONE lane)
            └─ worker: opencode run --dir wt/wsN --attach http://127.0.0.1:4096
                 one `worker` agent on branch swarm/wsN
            ship_ready: merge swarm/wsN → master → full suite + check → deploy
  • Worktrees: /root/trading/wt/ws1..ws4, each on branch swarm/wsN.
  • Server: opencode serve --port 4096 (workers attach to it; do NOT go headless).
  • Live app: project@trading (bun build/index.js, port 4102), fronted by caddy/cloudflared.
  • Worker count is persisted in .swarm-workers; SWARM_MAX only initialises it.
  • Hourly caps: QUEUE_MAX_SCOPE_PER_HOUR / QUEUE_MAX_WORK_PER_HOUR (18 each).

3. Root causes (in the order they stacked)

  1. Attach port mismatch. queue-watch.sh / swarm.sh defaulted the server to http://127.0.0.1:4100, but opencode.service listens on 4096. Every worker logged worker started headless … and ran as a standalone ~600–800 MiB Bun runtime instead of an attached ~280 MiB client.
  2. No resource ceiling on the queue-watch cgroup (MemoryMax=infinity, CPUQuota=infinity). Only the opencode server was capped.
  3. Effective concurrency 4 (SWARM_MAX=4, .swarm-workers pinned 4) on 2 vCPU, plus two interactive opencode --continue sessions.
  4. One uncommitted file froze ALL shipping. ship_ready refused to merge while git status --porcelain was non-empty. An uncommitted OrderEntryForm.svelte (which actually fixed a committed syntax error) + two stray test files held every merge for hours.
  5. Stranded branches → re-planning. Nothing merged, so finished work sat on swarm/wsN while the board read the rows pending; the planner re-issued them and workers rebuilt work that already existed.
  6. Claim → release race. The watcher claimed a batch, then its claim-release ran a few seconds later before workers spawned — no lane was "live", so the whole fresh batch looked orphaned and went back to pending (claimed 12 then released 11 in the same second).
  7. Conflict supervisor disabled (QUEUE_UNBLOCK=0): genuinely conflicted branches never landed (conflicted rows grew to 23).
  8. No watchdog, so the loop ran silently for hours.
  9. The deeper cause (see §5, §6): parallel agents on coupled files + a batch merge. All four lanes rewrote the same "focus-entry shell" and each wrote its own test pins; there was no merge order that made the suite green.

4. What was changed

4.1 systemd drop-ins (outside git — durable)

/etc/systemd/system/queue-watch.service.d/zz-limits.conf

[Unit]
StartLimitIntervalSec=0

[Service]
MemoryAccounting=yes
MemoryHigh=2800M
MemoryMax=3G
MemorySwapMax=256M
CPUAccounting=yes
CPUQuota=200%
CPUWeight=50
IOWeight=50
Restart=always
RestartSec=15

The memory ceiling is the anti-OOM guard (the 2026-09-27 OOM was 5.6 GiB). CPU is allowed to use both cores; CPUWeight=50 keeps the trading app and sshd ahead under contention. RestartSec=15 stops the 5 s crash-spin seen at 16:44.

/etc/systemd/system/queue-watch.service.d/pareto-caps.conf (current)

[Service]
Environment=SWARM_MAX=3
Environment=SWARM_SHIP_AREAS=2
Environment=SWARM_SHIP_EVERY=600
Environment=SWARM_ATTACH=1
Environment=SWARM_SERVER=http://127.0.0.1:4096
Environment=QUEUE_SERVER=http://127.0.0.1:4096
Environment=STALL_MIN=5
Environment=QUEUE_MAX_SCOPE_PER_HOUR=18
Environment=QUEUE_MAX_WORK_PER_HOUR=18
Environment=QUEUE_UNBLOCK=1
Environment=QUEUE_UNBLOCK_MAX_TRIES=3
Environment=QUEUE_WATCH_SLEEP=30
Environment=QUEUE_WATCH_REFILL_SLEEP=5

Apply: systemctl daemon-reload && systemctl restart queue-watch.

4.2 scripts/queue-watch.sh

  • Default server port 4100 → 4096.
  • Skip claim-release while a round is active (elif round_alive; then …) — kills the claim→release→re-claim churn. The release still runs between rounds.
  • Master-must-be-green guard (§7) before enqueuing new work.

4.3 scripts/swarm.sh

  • Default server port 4100 → 4096.
  • Ship-hold scoped to overlapping files. It holds only when a candidate branch actually touches a file that is dirty/untracked on master (matches if [ "$overlap" = "1" ]), instead of any stray file freezing every merge.
  • Rebase-on-start (§7): a lane carrying failed unmerged work is reset to master and rebuilt clean.

4.4 scripts/queue-doctor.sh (new) + cron

Circuit breaker. Flags uncommitted non-board files, ≥QUEUE_DOCTOR_CONFLICT_MAX (6) conflicted rows, branches ahead of master, or ≥QUEUE_DOCTOR_NOSHIP_MAX (3) consecutive no-ship rounds. It excludes docs/QUEUE.md (the line rewrites it every tick; the ledger commits it) so it does not cry wolf. Can auto-pause with QUEUE_DOCTOR_AUTOPAUSE=1. Installed at /etc/cron.d/queue-doctor (every 10 min):

*/10 * * * * root bash /root/trading/scripts/queue-doctor.sh check >> /root/trading/screenshots/queue-doctor.log 2>&1

4.5 One-off recovery commands

  • bash scripts/apply-db.sh — drizzle-kit push with stdin closed (aborts on destructive), restores the evt dedupe index and WAL. Missing migrations were the cause of 86/87 "merge" test failures (no such column: loom_width, no such table: order_change).
  • bun scripts/seed-users.ts — without it, every browser test throws HTTP 403 … Seed the logins first (a setup failure, not a code failure).

5. The focus-entry reconciliation (2026-09-30 ~21:00)

The four branches were divergent copies of one UI shell. Once the focus-entry shell was chosen as canonical, master = base + swarm/ws1 + swarm/ws3 + swarm/ws4, and 9 test files were reconciled to the shell (commit 1272a6a):

File Change
tests/fwd-orders.test.ts C160 next-step links now carry the order (?orderId=/?weaverOrderId=)
tests/c134-…shortname…test.ts summary is built from the C149 field spec; no Nickname row
tests/booking-fields.test.ts C47 matches deliveryTerm as a whole word (allows display-only o.deliveryTerms)
tests/booking-contract.test.ts the bookings list now shows/attaches the order (NEED-HOOK resolved)
tests/form-guide.test.ts checking zone = reviewGroups + auto: badge
tests/chip-suggest.test.ts C147 removed the production-term chips/datalists/dharas from OrderEntryForm
tests/c142-…all-order-forms.test.ts rendered picks row is the shell's label+input
tests/c143-…in-field-box.test.ts added C146 fields (loomWidth/panna), 15 hidden carries, wizard one-step-at-a-time
tests/c144-…popups.test.ts items-start pt-2 (card pins to top), const wizScrollIn

Result: 3261 pass / 0 fail. This was the fix that let lanes actually ship again.

Caveat: commit f0ae6f6 (early in the incident) committed leaked working-tree rewrites of tests/c143-* / tests/c144-* alongside the needed OrderEntryForm syntax fix. Diff against 731efda if ever in doubt.

6. The robust fix — parallel only across file-disjoint areas

Root cause of the tangle: parallel agents editing the same coupled files, then a batch merge — that cannot converge. The fix is to parallelize only work the planner can prove disjoint:

  • One lane per overlapping cluster, up to 3 in parallel. SWARM_MAX=3, .swarm-workers = "3 0 0 …". The planner groups rows into workstreams that are file-disjoint by construction (rows that share a file land in the same lane and run sequentially); those workstreams run in parallel. A batch that is all one area (overlapping files) still runs as a single lane — correctly.
  • Ship: SWARM_SHIP_AREAS=2, SWARM_SHIP_EVERY=600 — a couple of ready lanes batch into one suite+deploy, so we don't pay ~8 min of tests per lane.
  • Do not raise lanes to brute-force speed when the planner returns one workstream: that means the work overlaps, and parallel lanes will conflict.
  • Rebase-on-start: run_workstream resets a lane to master when it carries unmerged commits whose last state was gatefail/fail; a lane whose state was ok/noop is left to ship.
  • Kept the valuable half: queue/planner, per-row gate, heavy.sh, apply-db.sh, deploy.sh, unblock.sh, the doctor.

Never again: ≥2 lanes on a shared refactor. If parallelism is ever wanted, gate it behind a hard file-lock (two concurrent rows share ZERO files), ≤2 lanes on this box, per-lane merge+gate.

7. Guardrails (what keeps it healthy)

  1. Master-must-be-green guard (queue-watch.sh): refuses to enqueue new work unless the current HEAD holds a verified-green marker ($(git rev-parse --git-common-dir)/swarm-suite-green). It verifies once per code change and backs off on red. Board/ledger-only commits do NOT invalidate the marker (otherwise the ledger auto-commit re-ran the suite every round).
  2. Rebase-on-start (swarm.sh): no divergent branches accumulate.
  3. queue-doctor (10-min cron): the tripwire; alerts (and can auto-pause).
  4. unblocker (QUEUE_UNBLOCK=1): lands conflicted/stuck rows by opening the real page and merging the branch.
  5. Caps (zz-limits.conf): memory can never OOM the host.

8. Diagnose it fast (copy-paste)

cd /root/trading

# One line: is the line stuck?
bash scripts/queue-doctor.sh status

# Services
systemctl status queue-watch opencode project@trading --no-pager | head -40

# Resources now + history
free -h; cat /proc/loadavg
sar -r 2>/dev/null | tail -20      # memory history
sar -q 2>/dev/null | tail -20      # load history

# What is eating the box
ps -eo pid,ppid,%cpu,%mem,rss,etime,cmd --sort=-%cpu | head
systemd-cgtop -n1 -m -1 | head
cat /sys/fs/cgroup/system.slice/queue-watch.service/memory.max

# The line's own story
tail -60 screenshots/queue-watch.log
grep -E "ship HELD|MERGE CONFLICT|already-satisfied|released .*abandoned|not verified green" screenshots/queue-watch.log | tail

# Board + branches
bun scripts/queue-next.ts --list | grep -E "conflicted|doing" | head
git status --short
for b in swarm/ws1 swarm/ws2 swarm/ws3 swarm/ws4; do echo "$b ahead=$(git rev-list --count master..$b)"; done
ls .queue-conflicts/ 2>/dev/null && cat .queue-conflicts/*.json 2>/dev/null

# Is HEAD verified green?
echo "marker=$(cat "$(git rev-parse --git-common-dir)/swarm-suite-green") HEAD=$(git rev-parse HEAD)"

Tell-tale signs

Log line Meaning
worker started headless port/auth wrong → workers not attaching (check SWARM_SERVER vs the real port)
ship HELD — a candidate branch touches a file… a lane edits a file dirty on master; commit/stash it
worker reported N row(s) already-satisfied … shipping the branch instead finished work stranded on an unmerged branch
released N abandoned claim(s) right after claimed N the claim/release race (should be gone)
not verified green — running the suite the green guard is verifying master before new work
conflicted rows + .queue-conflicts/*.json branches conflict; the supervisor must land them
another swarm run is active — skipping a second round was launched; the flock stops it (harmless)

9. Recovery procedure

  1. Stop the bleeding: bash scripts/queue-doctor.sh pause (or rm .queue-auto). systemctl stop queue-watch if you need the box quiet.
  2. Check for OOM: journalctl -k --since "1 hour ago" | grep -i oom.
  3. Land stranded work the system's way:
    • QUEUE_UNBLOCK=1, then bash scripts/unblock.sh status / once (one row at a time; it opens the page to verify).
    • cleanly-mergeable branches: git merge-tree --write-tree master swarm/wsN (rc=0 = clean) then let ship_ready do merge+gate+deploy.
  4. If you must merge by hand: merge one branch → bash scripts/apply-db.sh → bun run check + bun --smol test → keep only if green. Never merge all at once — the focus-entry branches overlap and produce semantic conflicts.
  5. If master is red: the guard will refuse new work. Green it (fix tests or revert the offending merge) and set the marker: git rev-parse HEAD > "$(git rev-parse --git-common-dir)/swarm-suite-green".
  6. Resume: systemctl start queue-watch (re-arms .queue-auto).

10. Never-again rules

  • Parallel only across file-disjoint workstreams (the planner's guarantee), not by brute-force lane count. One overlapping area = one lane.
  • Memory is the limit, not CPU. Keep MemoryMax on the swarm cgroup; CPU at ~100% while working is fine; swap thrash + OOM is not.
  • Rebase each lane to master at start so divergence cannot accrue.
  • Master must be green before new work (the guard). A red suite means nothing can ship, so the line would only re-work rows.
  • Never let one dirty file freeze the merge (the hold is scoped now).
  • Apply migrations as part of every merge (apply-db.sh) and seed users (seed-users.ts) before page-gate tests.
  • Keep the conflict supervisor on (QUEUE_UNBLOCK=1).
  • Watch it: the 10-minute doctor cron; wire scripts/notify.sh to a channel you read.

11. File reference

Path Purpose
/etc/systemd/system/queue-watch.service.d/{zz-limits,pareto-caps}.conf caps, ports, worker count, serial mode, supervisor
/etc/cron.d/queue-doctor 10-minute circuit breaker
/root/trading/scripts/queue-watch.sh watcher / claim-release / master-green guard / round launch
/root/trading/scripts/swarm.sh lane run, rebase-on-start, ship/merge
/root/trading/scripts/queue-doctor.sh doctor / pause
/root/trading/scripts/unblock.sh stuck-row / conflict supervisor
/root/trading/scripts/apply-db.sh, deploy.sh, heavy.sh, seed-users.ts schema, deploy, serialised gates, logins
/root/trading/screenshots/queue-watch.log the line's story
/root/trading/screenshots/swarm.log worker summaries / ship decisions
/root/trading/.swarm-workers persisted lane count (1 0 0 serial …)
$(git-common-dir)/swarm-suite-green verified-green suite marker

Appendix A — timeline (2026-09-30)

  • 16:10–16:50 load 36→72→84, mem ~3.5 GiB, queue-watch crash-looping every 5 s. 17:29 reboot.
  • ~18:05 discovered the headless-port bug; found 5 GB+ / 96% pattern; installed caps + enabled the doctor; fixed the port.
  • 18:39 observed the claim→release race (claimed 12 → released 11).
  • 19:00–20:00 found the stranded swarm/ws1..ws4 branches and the red suite; attempted merges → semantic conflicts; ran the unblocker (it landed C155 and correctly identified the red gate as the blocker).
  • 20:15 preserved experiments (wip/*), decided the focus-entry shell is canonical.
  • 20:30–21:00 reconciled the 9 stale test files → suite green (3261/0).
  • 21:30 implemented SERIAL mode + rebase-on-start + master-green guard; verified a single lane.
  • 21:50 doctor healthy, master clean, one lane running.

Appendix B — preserved branches & backups

  • wip/all-merged (6159464) — ws1+ws3+ws4 merged (shows the c45 regression).
  • wip/ws3-only (33b6dbc) — master+ws3 (near-best merge).
  • wip/manual-merge (b209110) — earlier hand-merge reference.
  • wip/ws2-divergent — the old divergent swarm/ws2 (superseded by the clean rebuild).
  • /root/queue-fix-backup-20260930-181707/ — pre-change copies of the scripts/drop-in.
  • /root/queue-wip-backup-20260930-183552/ — leaked-master-edit copies (c153, order page).