ralph

Imported from GiorgioRicciardiello/LabBrain (core/.claude/skills/ralph/SKILL.md). Install upstream with npx skills add GiorgioRicciardiello/LabBrain --skill ralph. Copyright stays with the author.

EXECUTE NOW

Target: $ARGUMENTS

Parse arguments:

• N (optional): task count. Omitted = show overview (Steps 0-3), ask before Step 4.
• --serial: force serial (one at a time). Default is parallel within each phase.
• --loop: one phase-aware batch, then output next command for fresh session.
• --batch [id]: specific batch only
• --type [type]: specific current_phase (reduce, create, reflect, reweave, verify, enrich). Extraction tasks: type: extract but current_phase: reduce -- use --type reduce.
• --dry-run: show plan without executing
• --handoff: emit HANDOFF block at end
• --verbose: full per-task output (default: quiet one-liner)
• --workers N: override max_parallel_workers. Never exceeds config. Ignored with --serial.
• --unblock: show failed tasks, offer retry/skip
• --retry TASK_ID: reset failed task to pending

Session setup (generate once per /ralph run):

VAULT_PATH="$(pwd)"
RALPH_SESSION="ralph-$(date +%Y%m%d-%H%M%S)"
NEIGHBORHOOD_BATCH="nb-$(date +%Y%m%d-%H%M%S)-$(openssl rand -hex 3)"

Timing Reset: START_MS and BATCH_START_MS MUST be set immediately before each dispatch (serial: line 339, parallel: line 244). Never reuse a timestamp from a prior task or session. record_perf() will reject durations <= 0 or > 60 minutes as corrupt.

CLI shorthand (used throughout): $QQ = cd _code && uv run python -m chronicle_r.queue_query "$VAULT_PATH"

START NOW.

Step -1: Unblock / Retry Handling

If --unblock:

$QQ alerts

For each failed task in the reconcilable array:

• Reconcilable: offer Supersede ($QQ supersede TASK_ID --by SUPERSEDING_BATCH) or Skip.
• Not reconcilable: offer Retry ($QQ retry TASK_ID) or Skip.
• All reconcilable: offer batch supersede.

STOP after displaying.

If --retry TASK_ID: Run $QQ retry TASK_ID. If limit reached: $QQ retry TASK_ID --force. STOP.

Session Batch Cap (Resource Guard)

Read processing.ralph from ops/config.yaml -- single source of truth. Never hardcode defaults.

grep -A 12 'ralph:' ops/config.yaml

Config contains max_parallel_workers and phase_caps (per phase). Use exactly as read. Missing key = warn, not default.

Phase-aware cap: --type set → use phase_caps[type]. Mixed → lowest applicable cap. If N > cap, clamp and warn:

[Resource guard] Clamped N from {requested} to {cap} (limit: {reason}).
Large batches exhaust session memory. Use --loop for large backlogs.
For auto-iteration: cd _code && uv run engram-ralph
To adjust caps: edit processing.ralph.phase_caps in ops/config.yaml

EFFECTIVE_WORKERS: --workers N → trust caller. Otherwise → max_parallel_workers. --serial → ignored.

Loop Mode (--loop)

When --loop is set:

1. Auto-compute N from phase-aware cap (user N ignored).
2. Run Steps 0-7 normally.
3. If pending > 0, append:

--=={ loop }==--
Batch complete: {processed} tasks in this session
Remaining: {pending} tasks ({phase breakdown}) | {steps} steps to finish
Next batch size: {next_cap} (based on {dominant_phase} phase cap)
Estimated sessions remaining: {ceil(pending / next_cap)}

To auto-iterate across sessions (recommended):
  cd _code && uv run engram-ralph
Or manually: /ralph --loop
Or specific phase: /ralph --loop --type {phase}

4. If pending == 0: --=={ loop complete }==-- All tasks processed.

Key: One batch per session max. Auto-iterate via cd _code && uv run engram-ralph (new claude -p per batch, logs to ops/daemon/logs/ralph-loop-*.log). Combines with --type, --batch, --verbose, --serial. Default: parallel except reduce (serial).

Step 0: Queue Health Advisory

0a. Phase Ordering Tips

ADVISOR=$(cd _code && uv run python -m chronicle_r.vault_advisor "$VAULT_PATH" \
    --context ralph --include-phase-tips --max 4 2>/dev/null)

Store tips for Step 3. Do NOT display separately. Failures → proceed silently.

0b. Content Depth Advisory

• Stub sources (content_depth: stub): warn, recommend batch-failing:

  [Stub Source Warning] {N} source(s) are stubs (metadata only).
  Recommend batch-failing before processing.
    -> Batch-fail all stubs? (y/n)
    -> To fetch content first: /learn {title} for each, then /seed again
  

  Confirmed → mark failed with "zero-claim extraction: stub source with no extractable content".

• Abstract-only (content_depth: abstract): informational:

  [Content Depth Advisory] {N} source(s) at abstract scope.
  Extraction limited to claims/evidence/questions (no methods or design patterns).
  

MANDATORY: SUBAGENT SPAWNING

You MUST use the Agent tool for EVERY task. No exceptions. The lead session's ONLY job is: read queue, spawn subagent, evaluate return, update queue, repeat.

Phase Configuration

See _shared/ralph-phase-config.md for phase-to-agent mapping (model, maxTurns, rationale).

Steps 1-3: Read, Filter, Overview

Step 1: Read Queue State

$QQ stats                    # total, by_status, pending_by_phase, by_phase
$QQ steps                    # total_steps, pending_tasks, steps_by_current_phase
$QQ actionable [--limit N] [--type PHASE] [--batch ID]  # actionable list + blocked
$QQ tasks --limit N --siblings [--type PHASE] [--batch ID]  # full metadata + siblings
$QQ siblings TASK_ID         # siblings for one task

No queue file → "Queue is empty. Use /seed to add sources."

Step 2: Filter Tasks

CLI handles filtering, phase gating, batch grouping. Phase gate:

• Any pending at create/enrich → reflect/reweave blocked.
• Any pending at reflect → reweave blocked.
• reduce/verify → never blocked.

Use CLI output directly -- do NOT re-implement filtering.

Phase order: claim = create→reflect→reweave→verify. Enrichment = enrich→reflect→reweave→verify.

Step 3: Queue Overview

STATS=$($QQ stats)
ACTIONABLE=$($QQ actionable --limit 8)
ALERTS=$($QQ alerts)

Canonical pipeline order: reduce, create, enrich, reflect, reweave, verify.

--=={ ralph }==--

Queue: X pending ({phase breakdown}) | {steps} steps to finish
Done: Y | Failed: Z
Actionable: {count} ({N} reduce, {N} create, {N} enrich, ...)
Blocked: {N} reflect, {N} reweave -- {short reason}
Failed: {N} tasks ({N} at retry limit) -- /ralph --unblock to review

Next:
1. {id} -- {current_phase} -- {target (~60 chars)}
...

Options:
- cd _code && uv run engram-ralph -- auto-iterate across sessions
- /ralph {N} --type reduce -- process extractions first
- /ralph {N} --type enrich -- clear enrich backlog (unlocks reflect)
- /ralph {N} -- all eligible tasks
- /ralph --unblock -- review and retry failed tasks

Formatting rules:

• Actionable: group by current_phase in PIPELINE_ORDER. Drop zero-count. Extraction tasks (type: extract, current_phase: reduce) → label as reduce.
• Blocked: only if gate-blocked. One-clause rationale. Omit if none.
• Failed: only if > 0. Omit otherwise.
• Next: up to 8, sorted by PIPELINE_ORDER then queue position. Titles ~60 chars.
• Options: engram-ralph first, then 2-4 commands by PIPELINE_ORDER (earliest = strategic).

All blocked: show queue state + Suggested: /ralph {M} --type {earlier_phase}. --dry-run: STOP here. N provided: Continue to Step 4 (default parallel; --serial for serial). Bare /ralph: Ask user, wait for answer.

Step 4: Process Loop

Dispatch: --serial OR reduce phase → Step 4-serial. Otherwise → Step 4-parallel.

Reduce always serial (each extraction creates new queue entries).

Step 4-parallel: Parallel Dispatch

All N tasks at same phase (enforced by gate).

4p-0. Batch Pre-flight (reflect/reweave only)

cd _code && uv run qmd update && uv run qmd embed

Failure → log "Index update skipped" and continue. Non-blocking.

4p-a. Build All Prompts

Read tasks: $QQ tasks --limit N --siblings [--type PHASE] [--batch ID]

Build prompts per 4b rules below. Reflect/reweave: prepend PARALLEL_MODE: true. Enrich: group by target — shared targets serialize after batch.

4p-b. Bulk Section Audit (Idempotency Guard)

Per-task: run $QQ check-completion TASK_ID. If completed: true or section_filled: true → collect for bulk advance via $QQ advance-batch TASK_ID_1 TASK_ID_2 .... Log: [preflight] {N} tasks pre-advanced (sections already filled). Only dispatch tasks needing work. Do NOT call record-perf for skipped tasks — they had no agent dispatch and would record zeros.

4p-c. Dispatch All Agents

BATCH_START_MS=$(( $(date +%s) * 1000 ))

N > EFFECTIVE_WORKERS → dispatch in waves. Otherwise → all at once.

Agent(
  subagent_type = "ralph-{current_phase}",
  prompt = {prompt with PARALLEL_MODE if reflect/reweave},
  description = "{current_phase}: {short target}",
  run_in_background = true
)

4p-d. Collect Results

As each agent completes, follow Shared Phase Evaluation below. Timing: ELAPSED_MS = END_MS - BATCH_START_MS (batch-relative, not isolated). Do not sum parallel durations. Skip record-perf for idempotency-skipped tasks (they were never dispatched).

4p-e. Batch Advance

$QQ advance-batch TASK_ID_1 TASK_ID_2 ... (successful + idempotency-skipped).

4p-f. Merge Sidecars (reflect/reweave only)

Spawn merge agent for ops/queue/results/*-sidecar.json:

Agent(
  subagent_type = "ralph-merge",
  prompt = "Apply sidecar files from parallel batch.
Sidecar directory: ops/queue/results/
Phase: {current_phase}
Tasks processed: {task IDs}

Read *-sidecar.json, apply topic map additions (reflect) or backward links (reweave), deduplicate, validate, archive sidecars.",
  description = "merge: {current_phase} sidecars"
)

4p-g. Enrich Collisions

Held-back enrich tasks → process serially via Step 4-serial.

4p-h. Report Progress

[batch] {current_phase}: {success}/{total} OK, {fail} failed, {skip} idempotency-skip
[merge] {topic_maps} topic maps, {links} backward links  (if merged)

Step 4-serial: Process Loop (Serial Mode)

--serial or reduce phase. Process up to N tasks.

4a. Select Next Task

Pick first pending. Read: id, type, file, target, batch, current_phase, completed_phases.

Idempotency guard: Phase section already filled → skip, $QQ advance TASK_ID. Log skip.

Verbose mode only: print task header. Quiet (default): one-liner in 4e suffices.

4b. Build Subagent Prompt

$QQ tasks --limit N --siblings [--type PHASE] [--batch ID]

Common prompt header (every phase):

Task {ID} | Phase: {current_phase} | Target: {TARGET}
Task file: ops/queue/{FILE}
ONE PHASE ONLY.

Agents are self-contained. Do NOT tell subagents to invoke skills.

Phase-specific additions:

Reflect/reweave sibling block:

OTHER CLAIMS FROM THIS BATCH:
{verified sibling list, or "None yet"}

4c. Spawn Named Subagent (MANDATORY)

START_MS=$(( $(date +%s) * 1000 ))

Agent(
  subagent_type = "ralph-{current_phase}",
  prompt = {from 4b},
  description = "{current_phase}: {short target}"
)

You MUST call the Agent tool. Never execute inline.

4d. Evaluate + Advance

Follow Shared Phase Evaluation below. Use START_MS for timing.

4e. Report Progress (Quiet Default)

[{i}/{N}] {current_phase}: {target (~60 chars)} -- {OK|FAILED (reason)|SKIP (idempotency)}

Buffer learnings for final report. Verbose only with --verbose.

4f. Re-filter Tasks

Re-read ONLY when needed: after extract tasks, at phase boundaries, every 10th task. Homogeneous batches → skip re-reads.

Batch advance optimization: same-phase successes → $QQ advance-batch ID1 ID2 ...

Shared Phase Evaluation

Referenced by serial (4d) and parallel (4p-d). Run for each completing task.

1. Check completion:

$QQ check-completion TASK_ID

Returns: {"completed": bool, "note_found": bool, "actual_title": "...", "section_filled": bool}.

2. Success (completed=true): Parse HANDOFF for learnings. Proceed to advancement.

3. Failure (completed=false): Attempt recovery if ALL:

• Phase is reflect or reweave
• note_found: true
• No prior recovery this task this run

Recovery dispatch:

Agent(
  subagent_type = "ralph-{current_phase}",
  prompt = "RECOVERY_MODE: true
Task {ID} | Phase: {current_phase} | Target: {TARGET}
Task file: ops/queue/{FILE}

Previous agent exhausted turn budget. ONLY salvage existing work:
1. Read claim note for inline links already added.
2. Read task file for partial discovery trace.
3. Write ## /{current_phase} section summarizing what exists.
4. If PARALLEL_MODE active, write sidecar JSON.
5. Output HANDOFF block.
DO NOT run discovery searches. Finish in under 10 turns.",
  description = "recovery: {short target}"
)

Re-check. Still incomplete → $QQ fail TASK_ID --reason "subagent did not complete phase after recovery attempt".

4. Target sync: If actual_title ≠ queue target, update queue.json.

5. Post-create sync (MANDATORY): Parse Created: [[actual title]] from ## Create. Mismatch → update queue. Verify on disk → missing = mark failed: $QQ fail TASK_ID --reason "create phase completed but note not found on disk".

6. Post-reweave sync (MANDATORY): Check ## Reweave for | rename | row. Parse [[old]] -> [[new]]. Mismatch → update queue. Missing on disk → log warning.

7. Record perf:

END_MS=$(( $(date +%s) * 1000 ))
ELAPSED_MS=$((END_MS - START_MS))
$QQ record-perf TASK_ID --phase {phase} --duration-ms $ELAPSED_MS \
  --tokens {tokens or 0} --tool-uses {tool_uses or 0} --session $RALPH_SESSION

Recovery → add --recovery. Fire-and-forget.

8. Advance: $QQ advance TASK_ID (serial) or $QQ advance-batch ... (parallel, at end).

Extract tasks ONLY: Re-read queue after done (reduce creates new entries).

Step 5: Post-Batch Cross-Connect

$QQ batches --check-complete

Checks ALL batches retroactively. If complete_batches non-empty with 2+ claims:

1. Collect note paths from ## Create sections.
2. Spawn cross-connect:

Agent(
  subagent_type = "ralph-cross-connect",
  prompt = "Post-batch cross-connect for batch '{BATCH}'.
Notes: {titles + paths}
Verify sibling connections. Add missed links. Check backward gaps. Output HANDOFF.",
  description = "cross-connect: batch {BATCH}"
)

3. Parse HANDOFF, include in report.

Skip if: 1 claim or tasks still pending.

Step 6: Cleanup

Sidecar cleanup:

ls ops/queue/results/*-sidecar.json 2>/dev/null | wc -l

Orphans → [Warning] {N} orphaned sidecar files. Run ralph-merge or delete.

Neighborhood sidecars from prior runs are stale but NOT auto-deleted. Cleanup via /archive-batch or rm ops/queue/neighborhoods/*.json.

Topic Map GC (Step 6b):

cd _code && uv run python scripts/maintenance/gc_topic_maps.py "$VAULT_PATH" --fix --json

Dead links removed → [gc] Removed {N} dead link(s) from {M} topic map(s). Backups: .md.bak. Clean/failed → skip/warn. Non-blocking.

Step 7: Final Report

$QQ perf --group-by phase

--=={ ralph }==--

Processed: {count} tasks
  {phase breakdown}
  Recovery dispatches: {count} ({recovered} salvaged, {still_failed} failed)
  Failed this run: {count} (reasons: ...)

Subagents spawned: {count} (MUST equal tasks dispatched, excludes idempotency skips)

Performance (this batch):
  {phase}: avg {Xs}, avg {Nk} tokens ({count} tasks)
  Total: {Xs} wall-clock, {Nk} tokens

Learnings captured:
  {list or "None"}

Maintenance:
  {GC line if applicable, otherwise omit}

Queue state:
  Pending: {count} ({phase breakdown}) | {steps} steps to finish
  Done: {count} | Failed: {count} ({N} at retry limit)

Next steps:
  {pending → /ralph {N}; batch complete → /archive-batch; failed → /ralph --unblock; empty → done}

First-Queue-Clear Milestone (show once):

test -f ops/.queue-first-clear && echo "seen" || echo "new"

If new: First queue backlog cleared. Run /next for recommendations.

Slack notification:

cd _code && uv run python scripts/hooks/slack_ralph_hook.py \
  --vault "$VAULT_PATH" \
  --total {total} --completed {completed} --failed {failed} --skipped {skipped} \
  --phases '{phase_json}' --pending {pending} --done {done} --queue-failed {qfailed} \
  --elapsed {seconds}

Verification: Subagents spawned MUST = tasks processed. Mismatch = inline execution = error.

If --handoff: emit per _shared/pipeline-conventions.md ## Unified Handoff Protocol. Fields: context=ralph, phase=orchestration, action={processed summary}, next={recommendation}, learnings (include [Process gap]), notes={queue updates}.

Error Recovery

• Subagent crash: Queue retains current_phase. Re-run /ralph picks it up. Repeated crashes → failed (Gate 2). Use /ralph --unblock.
• Phase gate deadlock: $QQ alerts → mark stuck tasks failed ($QQ fail TASK_ID --reason "...") to unblock downstream.
• Retry exhaustion: 8-attempt limit → manual fix + force-retry, or archive.
• Queue corruption: Report and stop. No auto-fix.

Quality Gates & Constraints

Never: execute inline, >1 phase/subagent, auto-retry failed, skip advancement, process non-pending, run on bad queue, edit shared resources directly in parallel (use sidecars).

Always: spawn subagent for every task, include siblings in reflect/reweave, re-read after extract, log learnings, report failures, verify subagent count = task count.