Skip to content
OpenSmartRoute
Skillv1.0.0

joelclaw-system-check

Run comprehensive joelclaw health checks, including postboot/daily checks for NAS, MinIO, Convex, Postgres, Typesense, Inngest, Panda/Flagg authority split, k8s, worker, Redis, OTEL, tests, repo sync,

by joelhooks(0) 0 installs
Free
Sign in to install

Free account. Installing gives you the manifest plus copy-paste snippets.

See reviews

About

Imported from joelhooks/joelclaw (skills/joelclaw-system-check/SKILL.md). Install upstream with npx skills add joelhooks/joelclaw --skill joelclaw-system-check. Copyright stays with the author.

joelclaw System Health Check

Run scripts/postboot.sh for the read-only postboot/daily topology sweep. It checks the Flagg/Panda authority split, NAS mounts, NFS tuning, custom NAS MinIO, Central Postgres, local Convex, Typesense, Inngest, and shadow Central health without printing secrets. The daily automation also reviews recent Codex infra transcripts so completed work does not stay trapped in chat.

Migration caveat — 2026-07-09

The current health script is still too Panda-centric: it treats Panda-era k8s/PDS/NodePort surfaces as if they were the whole joelclaw system. That is no longer aligned with the Central migration vision.

Health needs to become lane-based:

  • Central lane: Flagg/Mac Studio native launchd services, Redis, Typesense, Inngest, system-bus worker, agent-mail, run capture, session NAS backup receipts, reboot-survivable state.
  • Relay lane: Panda-only leftovers such as iMessage/account-bound relays, legacy webhook/callback surfaces, and explicit decommission blockers.
  • Satellite lane: Blaine/other Machines' JOELCLAW_CENTRAL_URL, outbox backlog, fresh capture timestamps, and transcript backup freshness.

Until that rewrite lands, read red Panda/k8s/PDS checks as migration TODOs, not proof that Flagg Central is dead.

~/Code/joelhooks/joelclaw-runtime/skills/joelclaw-system-check/scripts/postboot.sh

Run scripts/health.sh for the older full system health report with 1-10 score. This is heavier because it includes tests and TypeScript.

~/Code/joelhooks/joelclaw-runtime/skills/joelclaw-system-check/scripts/health.sh

Postboot/Daily Check

Use this after a reboot, after storage/network work, and for the daily automation.

Default command:

POSTBOOT_NAS_BENCHMARK_MIB=1 \
  ~/Code/joelhooks/joelclaw-runtime/skills/joelclaw-system-check/scripts/postboot.sh

Default posture:

  • Read-only except for the tiny NAS verifier write probe under the already-designated s3 proof paths.
  • Does not run the Convex write/read/delete smoke unless POSTBOOT_HEAVY=1.
  • Does not run the legacy full health.sh unless POSTBOOT_RUN_FULL_HEALTH=1.
  • Never prints raw .env, MinIO keys, Convex admin keys, or Postgres URLs with passwords.
  • Treats custom NAS MinIO on 100.67.156.41:39000 as canonical.
  • Treats ASUSTOR MinIO CE on 29990 as a warning/reference surface only.
  • On Flagg, expects JOELCLAW_CENTRAL_URL and direct Typesense helpers to still point at Panda until explicit Central cutover.

Critical checks:

Check What Green Means
authority split env Flagg still points Central capture/search at Panda while Flagg is shadow
Inngest direct health the authoritative Inngest endpoint responds on /health; on Flagg this falls back to http://panda:8288 if INNGEST_URL is unset
Run capture health Panda /api/runs/health returns ok=true with local Machine auth
Typesense configured health configured Typesense endpoint responds on /health
NAS launchd label system/com.joelclaw.central.nas-mounts is loaded with no failing last exit
NAS route 10GbE/MTU route to 192.168.1.163 uses en0 and MTU 8192
NAS mounts status /Volumes/nas-nvme and /Volumes/three-body are mounted from LAN IP exports
NAS verifier write probe service checkout verify-nas.sh passes
NFS tuned options live mounts show rsize=524288,wsize=524288,readahead=128
custom MinIO ready/live custom NAS MinIO responds on 39000
Central Postgres socket, TCP, and readiness checks pass
local Convex backend /version and dashboard respond

Warning/reference checks:

  • custom MinIO console TCP on 39001
  • ASUSTOR MinIO CE readiness on 29990
  • aggregate joelclaw status and joelclaw inngest status CLI wrappers, because those can reveal local env drift even when the authoritative endpoint is healthy
  • Convex LAN and tailnet forwards
  • local Convex LAN forwarder LaunchAgent
  • Flagg shadow Central health
  • gated heavy checks

If postboot.sh fails, fix critical failures first. Do not chase warning/reference failures before the required path is green.

Recent Codex Transcript Review

The daily automation should review recent Codex infra sessions after the health script runs.

Preferred indexed search:

Use recall and the current session-search contract. Keep private queries in structured MCP arguments. Raw evidence requires a scope-bound evidenceDrilldownReceipt; missing projections do not grant raw access.

If indexed results are stale or missing, inspect metadata without reading transcript bodies. Raw content still requires the scoped receipt:

tail -n 80 ~/.codex/session_index.jsonl
find ~/.codex -type f -name "rollout-*.jsonl" -mtime -2

Review only bounded snippets, final answers, commands, and receipts. Do not dump full transcripts or secrets.

Promote durable facts into the right surface:

  • system topology and Panda/Flagg authority -> skills/system-architecture/SKILL.md
  • NAS access, LAN/MTU/NFS behavior -> skills/three-body/SKILL.md
  • custom NAS MinIO behavior -> skills/minio/SKILL.md
  • local Convex exposure and daily-use shape -> skills/local-convex/SKILL.md
  • dated receipts and decisions -> .brain/resources/*.svx

Treat in-progress subagent threads and sessions without a final answer as leads, not architecture truth.

Active Codex automation:

  • ID: daily-joelclaw-postboot-system-check
  • Schedule: daily at 08:15 local time
  • Workspace: /Users/joel/Code/joelhooks/joelclaw-runtime
  • Command intent: run POSTBOOT_NAS_BENCHMARK_MIB=1 skills/joelclaw-system-check/scripts/postboot.sh, summarize failures/warnings, review recent Codex infra transcripts, report uncaptured durable facts, and avoid secret output.

Full Health Check (16 components)

Check What Green (10) Yellow (5-7) Red (1-3)
k8s cluster pods in joelclaw namespace 4/4 Running, 0 restarts partial pods no pods
pds AT Proto PDS on :9627 version + collections pod running, host publish degraded pod not running
worker system-bus on :3111 16+ functions responding, low count down
inngest server :8288 reachable responding — down
agent-mail Flagg-local MCP mail on :8765 alive + mailbox visible degraded counts/search unavailable
redis/gateway Redis + gateway session queues connected, low pending queue connected, backlog rising unavailable
typesense/otel Typesense health + OTEL query path healthy + queryable healthy, query degraded unavailable
tests isolated per-file bun test in system-bus 0 fail — failures
tsc tsc --noEmit clean — type errors
repo sync monorepo HEAD vs origin/main in sync ahead/behind repo unavailable
memory pipeline joelclaw inngest memory-health healthy checks degraded checks failing checks
pi-tools extension deps installed all 3 deps — missing
git config user.name + email set set — missing
active loops joelclaw loop list queryable query degraded unavailable
gogcli Google Workspace auth account authed, token valid token stored, no password not configured
disk free space + loop tmp <80% used — >80%
stale tests __tests__/ + acceptance tests clean — present

Agent Session Capture + NAS Backup Check

Use the agent-session-capture-backup skill when the check needs proof that Pi/Claude/Codex transcript activity is captured and backed up across Flagg, Blaine, and Panda.

Fast audit/repair command from ~/Code/joelhooks/joelclaw:

bun scripts/agent-session-audit-backup.ts \
  --hosts flagg,blaine,panda \
  --central-url http://joels-mac-studio.tail7af24.ts.net:3111 \
  --backup-root /Volumes/three-body/sessions \
  --repair-env \
  --sync=true \
  --replay-outbox \
  --replay-limit 250 \
  --replay-max-bytes 10485760

Daily durable workflow:

  • function: system/agent-session.capture-backup.verify
  • event: system/agent-session.capture-backup.requested
  • cron: TZ=America/Los_Angeles 15 5 * * *
  • receipts: /Volumes/three-body/sessions/receipts/*.json

Treat stale or empty /Volumes/three-body/sessions as a critical backup failure even when runs_dev Typesense indexes are fresh. Typesense is derived; raw transcripts and run blobs are the source of truth.

When to Run

  • Session start — orient on system state before doing work
  • After loops complete — verify nothing broke
  • After infra changes — k8s, worker, Redis config
  • When something feels off — quick triage

Fixing Common Issues

Repo drift: cd ~/Code/joelhooks/joelclaw && git fetch origin && git status -sb

pi-tools broken: cd ~/.pi/agent/git/github.com/joelhooks/pi-tools && bun add @sinclair/typebox @mariozechner/pi-coding-agent @mariozechner/pi-tui @mariozechner/pi-ai

PDS unreachable: curl -fsS http://localhost:9627/xrpc/_health then kubectl get deploy,svc,pods,pvc -n joelclaw | rg 'bluesky-pds|NAME' (or if pod down: kubectl rollout restart deployment/bluesky-pds -n joelclaw)

Worker down: joelclaw inngest restart-worker --register

Failing tests: Inspect the affected behavior and repair valid coverage. Test provenance alone never authorizes removal.

System-bus test false reds: the health script runs each src/**/*.test.ts file in its own Bun process because several legacy tests monkey-patch globals or use mock.module. If the aggregate health check is green but raw bun test is red, suspect inter-file mock leakage before treating runtime code as broken.

Loop artifacts: Inspect task ownership and unharvested work. Remove only verified disposable artifacts within the authorized scope.

Inngest Hung-Run Quick Triage

When a run appears stuck after first step:

joelclaw run <run-id>

If trace shows Finalization failure with "Unable to reach SDK URL":

  1. Verify registration/health: joelclaw inngest status

  2. Verify function is present where expected: joelclaw functions | rg -i "manifest-archive|<function-name>"

  3. Check for stale app registrations in Inngest UI/API and remove stale SDK URLs.

  4. Assume possible handler blocking (not just network): review recent step code for filesystem/Redis/subprocess blocking before step response.

Use it

Copy one of these into your project. Installing also returns the manifest and these snippets.

yaml
targets:
  - https://api.opensmartroute.ai/api/v1/registry/joelhooks-joelclaw-joelclaw-system-check/manifest   # or paste the manifest below

Manifest

An Open Capability Manifest: the router reads it to know what this does, what it costs and when to pick it.

joelhooks-joelclaw-joelclaw-system-check.ocm.jsonjson
{
  "ocm": "1",
  "id": "joelhooks-joelclaw-joelclaw-system-check",
  "kind": "skill",
  "name": "joelclaw-system-check",
  "description": "Run comprehensive joelclaw health checks, including postboot/daily checks for NAS, MinIO, Convex, Postgres, Typesense, Inngest, Panda/Flagg authority split, k8s, worker, Redis, OTEL, tests, repo sync, memory pipeline, and disk. Use when: 'system health', 'postboot check', 'daily system check', 'health check', 'is everything working', 'system status', 'how's the system', 'check everything', or at session start to orient.",
  "publisher": "joelhooks",
  "version": "1.0.0",
  "capabilities": {
    "domains": [
      "coding"
    ],
    "tags": [
      "skill-md",
      "joelclaw",
      "health",
      "diagnostics",
      "checks",
      "operations",
      "skills-sh"
    ],
    "languages": [
      "en"
    ]
  },
  "quality_prior": 0.6,
  "examples": [
    "Run comprehensive joelclaw health checks, including postboot/daily checks for NAS, MinIO, Convex, Postgres, Typesense, Inngest, Panda/Flagg authority split, k8s, worker, Redis, OTEL, tests, repo sync, memory pipeline, and disk. Use when: 'system health', 'postboot check', 'daily system check', 'health check', 'is everything working', 'system status', 'how's the system', 'check everything', or at session start to orient."
  ],
  "primary": false,
  "metadata": {
    "source": {
      "provider": "skills.sh",
      "repository": "https://github.com/joelhooks/joelclaw",
      "path": "skills/joelclaw-system-check/SKILL.md",
      "ref": "HEAD",
      "url": "https://github.com/joelhooks/joelclaw/blob/HEAD/skills/joelclaw-system-check/SKILL.md",
      "key": "joelhooks/joelclaw/skills/joelclaw-system-check/SKILL.md"
    }
  },
  "instructions": "# joelclaw System Health Check\n\nRun `scripts/postboot.sh` for the read-only postboot/daily topology sweep. It checks the Flagg/Panda authority split, NAS mounts, NFS tuning, custom NAS MinIO, Central Postgres, local Convex, Typesense, Inngest, and shadow Central health without printing secrets. The daily automation also reviews recent Codex infra transcripts so completed work does not stay trapped in chat.\n\n## Migration caveat — 2026-07-09\n\nThe current health script is still too Panda-centric: it treats Panda-era k8s/PDS/NodePort surfaces as if they were the whole joelclaw system. That is no l",
  "cost": {
    "context_tokens": 2601
  }
}

Fetch it by URL: GET /api/v1/registry/joelhooks-joelclaw-joelclaw-system-check/manifest?version=1.0.0

Reviews

Star ratings from people who tried it. One review per account; edit yours any time.

No reviews yet. Install it, try it, and be the first to rate it.