Imported from karlorz/agent-skills (skills/host-backup-restore/skills/host-backup-restore/SKILL.md). Install upstream with npx skills add karlorz/agent-skills --skill host-backup-restore. Copyright stays with the author.

Host Backup Restore

Orchestrates host infrastructure backup and restore into a single flow: Caddy reverse-proxy domains, databases, systemd services, SSH configs, Hermes agent snapshots, and apt package lists.

Supports interactive (AskUserQuestion) mode (default), non-interactive CLI mode, backup profiles, post-discovery research, and skillwiki capture.

Quick Start

# Interactive backup — runs discover.sh, presents profile selection, then AskUserQuestion
/host-backup-restore sg01

# With a specific profile
/host-backup-restore sg01 --profile quick

# Non-interactive: use a preset profile
bash scripts/host-backup-cli.sh --host sg01 --profile full

# Non-interactive: quick backup (hermes + databases + base + caddy)
bash scripts/host-backup-cli.sh --host sg01 --profile quick

# Save a custom profile for reuse
bash scripts/host-backup-cli.sh --host sg01 --groups "hermes,databases,caddy_domains" --save-profile daily

# List all available profiles
bash scripts/host-backup-cli.sh --list-profiles

# Backup with post-discovery research
bash scripts/host-backup-cli.sh --host sg01 --profile full --research

# Portfolio Lab bundle retrieval (exclusive profile; retrieval + canonical verify)
# Obtain this lowercase SHA-256 from a separate trusted channel; it is not the
# retrieved .sha256 sidecar. --dest must be an absolute local path.
bash scripts/host-backup-cli.sh --profile portfolio-lab \
  --source-archive 'backup@storage.example:/mnt/encrypted-backups/portfolio-lab-20260814.portfolio-lab-recovery.tar' \
  --trusted-sha256 '<independently-trusted-archive-sha256>' \
  --dest "$HOME/Desktop/backups/portfolio-lab/"

# Portfolio Lab bundle restore to a dev target (explicit target mode; see below)
bash scripts/host-restore-cli.sh --archive ./portfolio-lab-20260814.portfolio-lab-recovery.tar \
  --trusted-sha256 '<independently-trusted-archive-sha256>' \
  --target devbox --user agent --target-mode dev --app-dir /srv/app --web-root /srv/www \
  --groups portfolio_lab

# Restore to a fresh host
bash scripts/host-restore-cli.sh --archive ./sg01-backup.tar.gz --target newhost --all

# Restore SSH + Tailscale identity after OS reinstall (explicit opt-in)
bash scripts/host-restore-cli.sh --archive ./sg02-backup.tar.gz --target sg02 --groups "ssh,tailscale" --restore-identity

Non-Root User Setup (Recommended)

By default, the generic host backup and restore operations use the non-root agent user for SSH. The Portfolio Lab recovery wrappers execute the packaged recovery CLI directly: use --user root for a production restore or activation preparation, and for a dev restore that requests --start-dev-api, because those commands write systemd paths and do not invoke sudo. A non-root user remains appropriate for retrieval and a staged-only dev restore.

Prerequisite: Bootstrap the agent user

Run once per target host to create the agent user with passwordless sudo:

bash scripts/hermes/setup-remote-user.sh <host>

This connects as root (one-time bootstrap), creates the agent user, grants passwordless sudo, deploys your SSH key, and optionally writes an SSH config alias.

How non-root works

The CLI scripts default to agent@<host> for SSH connections:

The agent user requires passwordless sudo for the generic host operations that invoke it. Portfolio Lab recovery commands do not use sudo; select --user root when the canonical operation requires systemd access.

Specifying a user explicitly

# Backup as root (if needed for system-level operations)
bash scripts/host-backup-cli.sh --host sg01 --user root --profile full

# Backup as a specific non-root user
bash scripts/host-backup-cli.sh --host sg01 --user deploy --profile quick

# Restore to a target as non-root agent (default)
bash scripts/host-restore-cli.sh --archive ./backup.tar.gz --target newhost --all

# Restore as root
bash scripts/host-restore-cli.sh --archive ./backup.tar.gz --target newhost --user root --all

# Using SSH config alias (handles user/key/port in ~/.ssh/config)
bash scripts/host-backup-cli.sh --host sg01-agent --profile full

Security

• After bootstrapping agent user, consider disabling root SSH login on the target host
• Passwordless sudo is required for non-interactive automation (systemd services, Caddy config)
• The agent user's SSH key is deployed from your local ~/.ssh/id_ed25519.pub or a specified key

Architecture

host-backup-restore/
├── SKILL.md                    # This file — interactive flow + orchestration
├── .claude-plugin/
│   ├── plugin.json             # Plugin manifest (v{{VERSION}})
│   └── agents/
│       ├── backup-worker.md    # Sonnet-pinned worker for general tasks
│       └── hermes-backup-worker.md  # Sonnet-pinned worker for Hermes ops
├── scripts/
│   ├── hermes/
│   │   ├── discover-hermes.sh      # Hermes-specific SSH discovery
│   │   ├── remote-backup.sh        # Remote Hermes backup orchestrator
│   │   ├── remote-restore.sh       # Remote Hermes restore via import
│   │   ├── pre-inspect.sh          # Restore target readiness check
│   │   ├── restore-validate.sh     # Post-restore Hermes validation
│   │   ├── prune-backups.sh        # Retention pruning
│   │   ├── setup-remote-user.sh    # Non-root user bootstrap
│   │   ├── setup-remote-cron.sh    # Automated backup cron
│   │   └── setup-nonroot-hermes.sh # Non-root Hermes installation
│   ├── discover.sh             # SSH discovery: parse Caddyfile, detect services
│   ├── backup-host.sh          # Mechanical backup script (reads manifest)
│   ├── host-backup-cli.sh      # Non-interactive CLI backup (profile-aware)
│   ├── host-restore-cli.sh     # Non-interactive CLI restore
│   ├── portfolio-lab-backup.sh # Portfolio Lab bundle retrieval (exclusive group)
│   ├── portfolio-lab-restore.sh# Portfolio Lab bundle restore (exclusive group)
│   ├── portfolio_lab_archive.py# Stdlib-tarfile safety gate: validate members, extract exact bootstrap
│   ├── profiles.sh             # Profile management (presets + YAML)
│   └── research-host.sh        # Post-discovery research query generator
└── tests/
    ├── test-restore.sh         # Per-component restore verification (27 assertions)
    └── test-portfolio-lab-profile.sh  # Hermetic portfolio-lab profile tests

Data flow

1. discover.sh connects via SSH, parses Caddyfile for domain→upstream mappings, detects running databases, finds sqlite files, enumerates systemd units. Outputs JSON manifest at /tmp/host-backup-{hostname}-manifest.json.
2. profiles.sh resolves profile name to groups + hermes-tier (built-in presets or user YAML).
3. Interactive (this SKILL.md): reads the manifest, presents profile/service selection to user via AskUserQuestion, then runs backup/restore based on selection.
4. CLI (host-backup-cli.sh / host-restore-cli.sh): reads manifest and accepts group selection via flags or --profile.
5. backup-host.sh reads manifest + selected groups, backs up each component to a tarball.
6. research-host.sh generates research queries from manifest for deep-research skill.
7. test-restore.sh validates functional correctness per component.

Profile System

Backup profiles define which groups to back up and what hermes-tier to use. Four built-in presets plus unlimited custom profiles.

Built-in Presets

portfolio-lab is a first-class built-in preset (not a YAML-only profile) and its name is reserved — user YAML can never override it, and --save-profile portfolio-lab is refused. It selects ONLY the portfolio_lab group; combining it with --groups, --all, or any generic group fails (selections are never silently overridden: --profile portfolio-lab with any --groups, or --groups portfolio_lab with any --profile, is rejected). It follows the authoritative Portfolio Lab recovery CLI contract (create / verify / restore / activate-prod) — see Portfolio Lab Profile.

Custom Profiles

Create ~/.config/host-backup-restore/profiles.yaml:

profiles:
  daily:
    groups: [hermes, databases, base, caddy_domains]
    hermes_tier: full
    description: "Daily backup of essential services"
  weekly-full:
    groups: [base, ssh, tailscale, caddy_domains, hermes, databases, other_services, apt, wiki]
    hermes_tier: full
    description: "Weekly full infrastructure backup"
  hermes-only:
    groups: [hermes]
    hermes_tier: minimal
    description: "Quick Hermes snapshot before upgrades"

CLI Profile Flags

Interactive Profile Selection

For full, quick, minimal, and custom profiles, present profile selection after discovery. Route portfolio-lab directly to the dedicated no-discovery bundle flow instead: backup uses --profile portfolio-lab; restore uses --groups portfolio_lab with an explicit target mode and paths.

{
  "question": "Which backup profile for <host>?",
  "header": "Profile",
  "options": [
    {"label": "full (Recommended)", "description": "All 9 groups — complete infrastructure backup with SSH identity and Tailscale state"},
    {"label": "quick", "description": "Essential state: Hermes, databases, Caddy, base (skips systemd + apt)"},
    {"label": "minimal", "description": "Hermes agent state only — fastest snapshot"},
    {"label": "portfolio-lab", "description": "Retrieve or restore the isolated Portfolio Lab recovery bundle without discovery"},
    {"label": "Custom", "description": "Select individual groups manually"}
  ]
}

If portfolio-lab is selected, collect the explicit archive/storage and target inputs, then run the dedicated flow in Portfolio Lab Profile. If "Custom" is selected, fall back to the per-group AskUserQuestion flow (Step 4b).

Portfolio Lab Profile

portfolio-lab is a first-class built-in profile (not a YAML-only profile) for Portfolio Lab application bundles. It selects exactly one group — portfolio_lab — and is exclusive: it cannot be combined with generic host/identity/Caddy/Hermes groups (base, ssh, tailscale, caddy_domains, hermes, databases, other_services, apt, wiki) or with --all; such combinations fail closed. Selections are never silently overridden (--profile portfolio-lab + --groups, or --groups portfolio_lab + any --profile, is rejected), and the built-in name portfolio-lab is reserved against YAML override.

The profile follows the authoritative Portfolio Lab recovery CLI contract. The skill never invents a backup subcommand or a --no-activate/--activate surface on that CLI. The canonical surface is:

• create --app-dir --web-root --tasker-service --archive --storage-encryption-attested
• verify --archive
• restore --archive --app-dir --web-root --target-mode dev|prod [--allow-production-paths] [--start-dev-api] [--tasker-service]
• activate-prod --app-dir --web-root --tasker-service --confirm-authoritative-activation --former-authority-confirmed-stopped LABEL

Archives must be plaintext files named *.portfolio-lab-recovery.tar (any other format or name is rejected), with a .tar.sha256 checksum sidecar, an embedded recovery-manifest.json, and the recovery bootstrap at tools/portfolio_lab_recovery.py. Before it runs that archive-provided bootstrap, the wrapper requires --trusted-sha256: a lowercase SHA-256 obtained through an independent trusted channel, such as a signed release record or an authenticated out-of-band handoff. It compares that value to the archive bytes as well as validating the co-retrieved .sha256 sidecar. The archive and sidecar are not sufficient to authenticate attacker-supplied content. The canonical CLI then verifies the full manifest and member digests.

Portfolio Lab backup (retrieval + canonical verify)

Backup is retrieval scope only: the skill retrieves an existing source archive from explicit remote/source storage into an explicit local --dest, then runs canonical verify. It does not create the source archive, makes no storage-encryption attestation, runs no managed-host discovery, and never auto-transfers to cloud storage. A user@host:path source is an explicit rsync connection to that source, rather than use of the generic --host workflow.

bash scripts/host-backup-cli.sh --profile portfolio-lab \
  --source-archive 'backup@storage.example:/mnt/encrypted-backups/portfolio-lab-20260814.portfolio-lab-recovery.tar' \
  --trusted-sha256 '<independently-trusted-archive-sha256>' \
  --dest "$HOME/Desktop/backups/portfolio-lab/"

Required flags (all fail closed when missing):

Flow (portfolio-lab-backup.sh):

1. rsync the archive and its .sha256 checksum sidecar into --dest with resumable partial transfer (--partial-dir=.rsync-partial).
2. Compare the retrieved archive bytes to both the sidecar and the required independently trusted --trusted-sha256; abort on either mismatch.
3. Safety gate (stdlib tarfile, portfolio_lab_archive.py): validate every member — no traversal/absolute/symlink/device paths; exact regular-file members recovery-manifest.json + tools/portfolio_lab_recovery.py — and extract ONLY the exact bootstrap to controlled staging (never full-extract). Unsafe/malformed archives fail closed before any canonical call.
4. Run the canonical verify --archive <dest>/<archive> with the packaged bootstrap.

Creating the source archive happens on the source host with the canonical CLI — out of this skill's scope. Run create as root, because it captures, stops, and restarts the source Tasker service through systemctl. PORTFOLIO_LAB_PROJECT_DIR selects the project directory for python_runtime.sh; it does not change recovery CLI flags or defaults. Invoke <repo>/scripts/python_runtime.sh <repo>/scripts/portfolio_lab_recovery.py from that explicit project directory and write to an absolute encrypted storage mount path:

PORTFOLIO_LAB_PROJECT_DIR=/path/to/portfolio-lab /path/to/portfolio-lab/scripts/python_runtime.sh \
  /path/to/portfolio-lab/scripts/portfolio_lab_recovery.py \
  create --app-dir /srv/app --web-root /srv/www --tasker-service portfolio-lab-tasker \
  --archive /mnt/encrypted-backups/portfolio-lab-YYYYMMDD.portfolio-lab-recovery.tar --storage-encryption-attested

Portfolio Lab restore

Restore requires an explicit --target-mode dev|prod, --app-dir, --web-root, and an independently trusted --trusted-sha256. It is a safe fresh-target flow: the recovery CLI is packaged in the archive, so no existing checkout is required on the target. Production restore requires --user root (or an approved equivalent wrapper) and refuses to mutate targets until the archived Tasker service is inactive.

# Dev target
bash scripts/host-restore-cli.sh --archive ./portfolio-lab-20260814.portfolio-lab-recovery.tar \
  --trusted-sha256 '<independently-trusted-archive-sha256>' \
  --target devbox --user agent --target-mode dev --app-dir /srv/app --web-root /srv/www \
  --groups portfolio_lab

# Prod target — stage only; target Tasker must be inactive and activation is manual-only.
bash scripts/host-restore-cli.sh --archive ./portfolio-lab-20260814.portfolio-lab-recovery.tar \
  --trusted-sha256 '<independently-trusted-archive-sha256>' \
  --target prodbox --user root --target-mode prod --app-dir /srv/app --web-root /srv/www \
  --tasker-service portfolio-lab-tasker --allow-production-paths --groups portfolio_lab

Optional canonical passthroughs: --tasker-service NAME, --allow-production-paths (prod), --start-dev-api (dev).

Flow (portfolio-lab-restore.sh):

1. rsync the archive + .tar.sha256 sidecar to the target (--partial-dir, resumable).
2. Compare the archive bytes on the target to both the sidecar and the independently trusted --trusted-sha256; abort on either mismatch.
3. Safety gate (stdlib tarfile, piped to the target): validate every member — no traversal/absolute/symlink/device paths; exact regular-file members recovery-manifest.json + tools/portfolio_lab_recovery.py — and extract ONLY the exact bootstrap into a fresh mktemp staging directory with mode 0700. The wrapper never full-extracts, so an unsafe/malformed archive cannot mutate app/web or any path outside staging before canonical verification.
4. Run canonical verify --archive on the target with the packaged bootstrap BEFORE restore; abort on failure.
5. Run the canonical restore with the packaged bootstrap: python3 tools/portfolio_lab_recovery.py restore --archive <remote> --app-dir <dir> --web-root <dir> --target-mode <dev|prod> [passthroughs].

Activation is never automatic. The wrapper only prints the canonical manual activate-prod command after restore; it never runs it. Successful restores retain their remote staging directory so this command can use the verified packaged bootstrap. Remove that directory only after the attended activation decision or drill review is complete:

ssh <target> 'rm -rf <remote staging>'

cd <remote staging>/tools && python3 portfolio_lab_recovery.py activate-prod \
  --app-dir <dir> --web-root <dir> --tasker-service <svc> \
  --confirm-authoritative-activation --former-authority-confirmed-stopped <LABEL>

Safety invariants (enforced): portfolio-lab restore never restores credentials, identities (ssh/tailscale), whole Caddy config/cert data, or agent state (hermes); --restore-identity is rejected with the portfolio_lab group; production activation is never automatic. No flag is silently ignored on the portfolio-lab paths: backup rejects irrelevant generic flags (--user, --hermes-tier, --redetect, --research, --save-profile, --db-user, --db-pass), restore forwards --user (target becomes user@host) and requires --trusted-sha256, rejects --db-user/--db-pass/--allow-cross-distro, and --dry-run performs no ssh/rsync calls at all.

Model-Aware Agents

The skill uses a sonnet-pinned worker agent for mechanical tasks, keeping the orchestrator (main session) for user interaction and decision-making.

Agent: backup-worker (model: sonnet)

Defined in agents/backup-worker.md. Handles:

• SSH discovery (discover.sh)
• Backup execution (backup-host.sh)
• Restore execution (host-restore-cli.sh)
• Post-restore validation (test-restore.sh)
• Profile resolution (profiles.sh)

Orchestration Flow

User Session (opus/inherit)
  ├── Interactive decisions (AskUserQuestion)
  ├── Profile selection
  ├── Post-discovery research (deep-research skill)
  ├── Skillwiki capture
  ├── Spawns backup-worker (sonnet)
  │     ├── discover.sh
  │     ├── backup-host.sh
  │     ├── host-restore-cli.sh
  │     └── test-restore.sh
  └── Spawns hermes-backup-worker (sonnet)
        ├── discover-hermes.sh
        ├── remote-backup.sh
        ├── remote-restore.sh
        ├── pre-inspect.sh
        ├── restore-validate.sh
        └── prune-backups.sh

When to spawn backup-worker:

• After user has made all decisions (profile, groups, mode)
• For the mechanical backup/restore execution
• For post-restore validation

When to stay in orchestrator:

• User interaction (AskUserQuestion)
• Profile design and custom profile management
• Deep research on detected services
• Skillwiki knowledge capture

Interactive Mode (Default)

Entry point: /host-backup-restore [host] [mode] [options]

Arguments:

• host — SSH hostname (e.g. sg01, sg03, ptcloud). Required.
• mode — backup (default) or restore. Optional.
• --profile NAME — Use a named profile. Optional.
• --redetect — Re-run discovery instead of using cached manifest.
• --dest PATH — Backup destination directory.
• --dry-run — Preview what would be backed up without doing it.
• --research — Run post-discovery research. Optional.

Step 1 — Run discovery

Run discover.sh to detect all services on the target host:

SCRIPT_DIR="$(dirname "$(realpath "$0")") 2>/dev/null || echo /path/to/skill"
bash "$SCRIPT_DIR/scripts/discover.sh" <host>

Discovery output is cached at /tmp/host-backup-{hostname}-manifest.json. Use --redetect to force re-run.

Model note: Spawn backup-worker agent for discovery to use sonnet for the SSH-heavy work.

Step 2 — Present discovered services

Read the manifest and present the detected services to the user with a table. Example for sg01:

Detected services on sg01:
- Caddy domains: mon.karldigi.dev, status.karldigi.dev, term.karldigi.dev, bot.karldigi.dev, star.karldigi.dev (5 total)
- Hermes: v0.13.0 at /root/.hermes
- Databases: sqlite files (/root/.hermes/state.db, etc.), [redis/postgres/mysql as detected]
- Systemd services: hermes-gateway, hermes-dashboard, caddy, filebrowser, obsidian, xvfb, [others]
- Apt sources: deb https://... (N sources)

Step 3 — Profile selection (AskUserQuestion)

Use AskUserQuestion with profile options:

{
  "question": "Which backup profile for <host>?",
  "header": "Profile",
  "options": [
    {"label": "full (Recommended)", "description": "All 9 groups — complete infrastructure backup with SSH identity and Tailscale state"},
    {"label": "quick", "description": "Essential state: Hermes, databases, Caddy, base (skips systemd + apt)"},
    {"label": "minimal", "description": "Hermes agent state only — fastest snapshot"},
    {"label": "portfolio-lab", "description": "Retrieve or restore the isolated Portfolio Lab recovery bundle without discovery"},
    {"label": "Custom", "description": "Select individual groups manually"}
  ]
}

If a --profile flag was passed, skip this step and use the specified profile.

Step 4a — Preset profile path (full/quick/minimal)

The portfolio-lab preset is handled by the dedicated no-discovery bundle path above; it does not invoke discover.sh or backup-host.sh.

Resolve the profile via profiles.sh and run backup with the resolved groups:

source "$SCRIPT_DIR/scripts/profiles.sh"
resolve_profile "<profile_name>"
bash "$SCRIPT_DIR/scripts/backup-host.sh" "$MANIFEST_FILE" $PROFILE_GROUPS

Step 4b — Custom / "Select individually" path

For each detected group, use AskUserQuestion (yes/no). Iterate through groups one at a time. Only ask about groups that have detected services:

{
  "question": "Back up <group_name>? (<brief_description_of_what_this_covers>)",
  "header": "Service group",
  "options": [
    {"label": "Yes", "description": "Include this group in the backup"},
    {"label": "No", "description": "Skip this group"}
  ]
}

Group order and descriptions:

After collecting all answers, run backup-host.sh with the selected groups:

bash "$SCRIPT_DIR/scripts/backup-host.sh" "$MANIFEST" <selected_group1> <selected_group2> ...

Offer to save the selection as a custom profile:

{
  "question": "Save this selection as a custom profile for future use?",
  "header": "Save profile",
  "options": [
    {"label": "Yes", "description": "Save as a named profile in ~/.config/host-backup-restore/profiles.yaml"},
    {"label": "No", "description": "Continue without saving"}
  ]
}

Step 4c — "Restore" path

1. List available backups: Check ~/Desktop/backups/<host>/ for existing archives.
2. AskUserQuestion to select archive:

{
  "question": "Which backup archive do you want to restore from?",
  "header": "Restore archive",
  "options": [
    {"label": "backup-20260510-143000.tar.gz (78M, 2026-05-10)", "description": "Full backup with 54 files"},
    {"label": "backup-20260509-120000.tar.gz (45M, 2026-05-09)", "description": "Partial backup"},
    {"label": "Custom path", "description": "Specify a different archive path"}
  ]
}

3. AskUserQuestion for groups to restore (same per-group yes/no pattern as Step 4b).

If ssh or tailscale identity groups were selected after an OS reinstall:

ssh and tailscale are reinstall-prep identity groups. They are restorable via CLI only with explicit --restore-identity because they replace host trust and tailnet identity:

bash scripts/host-restore-cli.sh \
  --archive ~/Desktop/backups/sg02/sg02-backup-YYYYMMDD-HHMMSS.tar.gz \
  --target sg02 \
  --groups "ssh,tailscale" \
  --restore-identity

Operational findings from the sg02 Debian 13 reinstall:

1. Host key changes are expected after reinstall. Clear stale local entries first: ssh-keygen -R <alias>; ssh-keygen -R <ip>, then reconnect with ssh -o StrictHostKeyChecking=accept-new <alias>.
2. Fresh Debian 13 may lack ca-certificates. Tailscale apt install can fail with certificate verify failed until apt-get install -y ca-certificates && update-ca-certificates runs.
3. Fresh minimal hosts may lack rsync and python3. Identity restore must not depend on them. Stream tarballs over SSH and use POSIX tools for validation.
4. Install Tailscale before restoring /var/lib/tailscale. Restore the saved apt source/keyring first, install tailscale, stop tailscaled, extract the saved state, then systemctl daemon-reload && systemctl enable --now tailscaled.
5. Validate SSH before restart. After extracting /etc/ssh, run /usr/sbin/sshd -t; if it fails, roll back from the safety tarball before restarting SSH.
6. Refresh known_hosts after restoring old SSH host keys. The server key reverts to the backup identity, so local clients that accepted the post-reinstall key will see another host-key change.

The restore CLI creates remote safety tarballs under /root/host-restore-safety-*-ssh and /root/host-restore-safety-*-tailscale before overwriting identity material.

If caddy_domains was selected and Caddy is not on the target:

Check if Caddy exists on the target:

ssh <target> "which caddy 2>/dev/null || echo MISSING"

If MISSING, prompt the user:

{
  "question": "Caddy is not installed on <target>. Should I install it before restoring Caddy config?",
  "header": "Caddy install",
  "options": [
    {"label": "Yes (Recommended)", "description": "Install caddy via apt-get on the target, then restore config and restart the service"},
    {"label": "No", "description": "Restore config files only — Caddy won't serve domains until manually installed"}
  ]
}

If yes, run: ssh <target> "sudo apt-get install -y caddy"

Check wiki S3 mount status on target:

ssh <target> "df -T ~/wiki 2>/dev/null | grep -q fuse.rclone && echo 'MOUNTED' || echo 'MISSING'"

If MISSING, check FUSE availability:

ssh <target> "test -c /dev/fuse && echo 'FUSE_OK' || echo 'NO_FUSE'"

If FUSE_OK, prompt:

{
  "question": "Wiki S3 mount is not active on <target>. Should I set it up?",
  "header": "Wiki mount",
  "options": [
    {"label": "Yes (Recommended)", "description": "Restore rclone.conf from backup and mount wiki at ~/wiki"},
    {"label": "No", "description": "Skip — wiki will not be available on this host until manually mounted"}
  ]
}

If yes, run the wiki restore group via: bash scripts/host-restore-cli.sh --archive <path> --target <host> --groups wiki

If NO_FUSE, inform the user with fix guidance:

FUSE is not available on this host. The wiki S3 mount cannot be set up.

Fix options:
1. **LXC template (best):** Add `features: fuse=1` to the PVE base template
2. **LXC per-container:** Set `fuse=1` on the container features in PVE
3. **tmpfiles.d:** Create `/etc/tmpfiles.d/fuse.conf` with `c /dev/fuse 0666 root root - 10:229`

After FUSE is available, re-run the restore with `--groups wiki`.

4. Run restore:

bash "$SCRIPT_DIR/scripts/host-restore-cli.sh" --archive <archive_path> --target <host> --groups <selected_groups>

Restore best practices (from vault research):

• Stop gateway before importing — ssh <host> "systemctl --user stop hermes-gateway.service" to avoid conflicts with running processes. ^[queries/hermes-backup-validation-restore-preinspection.md]
• Check distro compatibility — Restoring apt sources across different distros (Debian→Ubuntu) breaks apt. Always verify source/target OS match.
• SQLite WAL mode — Naively copying a SQLite DB with active WAL mode misses -wal/-shm files. The hermes backup command handles this correctly. For manual sqlite files, use .backup command.
• Post-restore validation — Run the test harness to verify:

bash "$SCRIPT_DIR/tests/test-restore.sh" --manifest /tmp/host-backup-<host>-manifest.json

Step 5 — Archiving

Tarball the backup directory:

cd "$(dirname "$BACKUP_DIR")"
tar czf "<host>-backup-$(date +%Y%m%d-%H%M%S).tar.gz" "$(basename "$BACKUP_DIR")"

Step 6 — Skillwiki Capture (Optional)

After backup completes, offer to capture the host infrastructure snapshot to skillwiki:

{
  "question": "Capture host infrastructure snapshot to skillwiki?",
  "header": "Wiki capture",
  "options": [
    {"label": "Yes", "description": "Write infrastructure snapshot to skillwiki vault as a typed-knowledge page"},
    {"label": "No", "description": "Skip wiki capture"}
  ]
}

If yes, use the wiki-add-task or wiki-crystallize skill to capture:

# Capture as a typed-knowledge page
skillwiki wiki-crystallize --type entity --title "Host: <hostname>" --content "
## Infrastructure Snapshot (<date>)

**Hostname:** <hostname>
**OS:** <os_id> <os_version>
**Caddy domains:** <domain_list>
**Hermes:** <version> at <home>
**Databases:** <db_summary>
**Systemd services:** <service_list>
**Apt sources:** <source_count> sources
**Profile used:** <profile_name>
**Backup archive:** <archive_path>
"

This creates a point-in-time record of host infrastructure that can be queried later for drift detection or disaster recovery reference.

Step 7 — Post-Discovery Research (Optional)

If --research flag is passed or user opts in, generate research queries and run deep-research:

bash "$SCRIPT_DIR/scripts/research-host.sh" "$MANIFEST_FILE" --output "/tmp/host-backup-${HOST}-research"

Then invoke the deep-research skill for high-priority queries:

{
  "question": "Run post-discovery research on detected services?",
  "header": "Research",
  "options": [
    {"label": "Yes", "description": "Research Hermes version, OS security advisories, database backup best practices"},
    {"label": "No", "description": "Skip research and proceed with backup"}
  ]
}

Research topics generated from manifest:

• Hermes version changelog and known issues
• OS security advisories for detected distro/version
• Database backup best practices for detected DB types
• Caddy reverse proxy security and performance recommendations

CLI Commands

host-backup-cli.sh

Non-interactive backup for automation/cron/scripting.

bash scripts/host-backup-cli.sh [options]

Hermes tier mapping:

• minimal → hermes backup --quick (config + state only)
• standard / full → hermes backup (no flags, full zip — handles SQLite WAL mode)

Important: hermes backup does NOT support --tier. Using --tier causes a silent error that produces no backup zip. Use --quick instead.

# Examples
bash scripts/host-backup-cli.sh --host sg01 --profile full
bash scripts/host-backup-cli.sh --host sg01 --profile quick --research
bash scripts/host-backup-cli.sh --host sg01 --groups "caddy_domains,hermes" --save-profile web-only
bash scripts/host-backup-cli.sh --host sg01 --all --hermes-tier minimal --dest ~/backups
bash scripts/host-backup-cli.sh --list-profiles

host-restore-cli.sh

Non-interactive restore from a backup archive.

bash scripts/host-restore-cli.sh [options]

# Examples
bash scripts/host-restore-cli.sh --archive ./sg01-backup.tar.gz --target newhost --all
bash scripts/host-restore-cli.sh --archive ./sg01-backup.tar.gz --target newhost --groups "caddy_domains,hermes" --dry-run

Discovery

Cached manifest at /tmp/host-backup-{hostname}-manifest.json. Use --redetect to re-run.

discover.sh connects via SSH and detects:

• Caddyfile (/etc/caddy/Caddyfile) — domain names and upstream targets (via caddy adapt + JSON extraction with legacy fallback)
• Systemctl — service states (systemctl is-active, systemctl list-units)
• Database sockets — postgres, mysql, redis, mongodb listener detection + database name enumeration
• File system — sqlite .db files, installed packages, apt sources
• Hermes — version, HERMES_HOME path
• OS release — distro ID and version (for restore compatibility)

Example output (from sg01)

{
  "hostname": "sg01",
  "timestamp": "2026-05-10T14:30:00Z",
  "caddy_domains": [
    {"domain": "mon.karldigi.dev", "upstream": "localhost:3000"},
    {"domain": "status.karldigi.dev", "upstream": "localhost:3001"},
    {"domain": "term.karldigi.dev", "upstream": "localhost:8080"},
    {"domain": "bot.karldigi.dev", "upstream": "localhost:7456"},
    {"domain": "star.karldigi.dev", "upstream": ""}
  ],
  "hermes": {"version": "0.13.0", "home": "/root/.hermes"},
  "databases": {"sqlite": ["/root/.hermes/state.db"], "redis": ["6379"]},
  "other_services": [
    "hermes-gateway", "hermes-dashboard", "caddy",
    "filebrowser", "obsidian", "xvfb",
    "cmux-execd", "cmux-proxy", "cmux-ide", "cmux-worker-daemon"
  ],
  "apt_sources": ["deb https://deb.debian.org/debian trixie main", "deb https://deb.debian.org/debian trixie-updates main"],
  "os": "debian",
  "os_version": "13"
}

Restore Tests

The test harness validates functional correctness per component.

# Run all tests
bash tests/test-restore.sh --manifest /tmp/manifest.json

# Test a specific group only
bash tests/test-restore.sh --manifest /tmp/manifest.json --group caddy_domains

Restore test matrix (27 assertions across 8 restore groups)

Backup-only groups ssh and tailscale are reinstall-prep artifacts. Restore is deliberately manual because reusing SSH host keys or Tailscale machine identity affects host trust and tailnet identity.

CLI identity restore is available only with explicit opt-in:

bash scripts/host-restore-cli.sh --archive ./backup.tar.gz --target <host> --groups "ssh,tailscale" --restore-identity

The identity restore path is intentionally dependency-light for fresh reinstall targets:

• Uses SSH tar streaming, not target-side rsync
• Installs ca-certificates before using the Tailscale apt repo
• Installs tailscale before restoring /var/lib/tailscale
• Validates sshd -t before restarting SSH
• Emits known_hosts refresh guidance after SSH host key reuse

Known limitations (source-side)

The following test "failures" are source-side edge cases, not restore bugs:

• DNS resolution failure for Caddy domains (no public DNS pointing at test host)
• Database connection refused (locked DB during backup)
• Stopped services (not running on source during discovery)

Hermes Integration

The skill integrates with hermes backup for Hermes-specific snapshots:

# Full backup (SQLite-safe, handles WAL mode)
hermes backup -o hermes-backup.zip

# Quick snapshot (config + state only)
hermes backup --quick

# Restore
hermes import hermes-backup.zip

Hermes restore validation

After restoring Hermes to a target host, run this post-restore validation sequence:

1. CLI health: hermes --version && hermes doctor
2. Systemd services: systemctl --user status hermes-gateway.service and sudo systemctl status hermes-dashboard.service
3. Health endpoint: curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8642/health (expect 200)
4. API verification: curl -s -H "Authorization: Bearer $KEY" http://127.0.0.1:8642/v1/models
5. Content check: du -sh ~/.hermes/state.db, ls ~/.hermes/skills/ | wc -l, cat ~/.hermes/cron/jobs.json

Important: Stop the gateway BEFORE importing: systemctl --user stop hermes-gateway.service ^[entities/hermes-backup-restore-guide.md] Also stop the dashboard if it's running as a system service: sudo systemctl stop hermes-dashboard.service

For full Hermes backup/restore reference, see the [[hermes-cli]] skill.

Hermes Module (scripts/hermes/)

The scripts/hermes/ directory contains Hermes-specific backup/restore scripts absorbed from the standalone hermes-remote-backup skill. These scripts handle the Hermes agent layer of host backup via the official Hermes CLI.

Hermes Backup Worker Agent

The hermes-backup-worker agent (model: sonnet) orchestrates Hermes-specific mechanical tasks. It is spawned by the orchestrator (main session) for:

• Backup flow: discover-hermes.sh → remote-backup.sh → prune-backups.sh
• Restore flow: pre-inspect.sh → remote-restore.sh → restore-validate.sh
• Setup flow: setup-remote-user.sh → setup-remote-cron.sh → setup-nonroot-hermes.sh

Spawn pattern in interactive mode:

After user selects "hermes" group:
  → Spawn hermes-backup-worker (sonnet)
  → Agent runs: discover-hermes.sh → remote-backup.sh
  → Agent returns result summary
  → Orchestrator continues with next group or wiki capture

Model specification: Per [[concepts/claude-code-agent-model-specification]], the model: sonnet is set in the agent frontmatter (agents/hermes-backup-worker.md), not in plugin.json or SKILL.md. The Agent tool parameter can override at spawn time but defaults to the agent file setting.

Primary entry point: The hermes-backup-worker agent is the recommended way to perform Hermes backup/restore operations. Use Agent(subagent_type="hermes-backup-worker", ...) instead of calling scripts/hermes/ scripts directly. The agent handles script selection, error handling, and result reporting.

Performance note: hermes backup on sg01 creates a ~2.2 GB zip via SSH. The transfer can take 10+ minutes over WAN. Consider:

• Spawning hermes-backup-worker with run_in_background: true for non-blocking backup
• Using --profile minimal or --hermes-tier minimal for faster snapshots
• Running hermes backup directly on the host (ssh sg01 "hermes backup -o backup.zip") for large transfers

DevSH Testing

For automated restore testing on ephemeral VMs:

# Create a devsh VM (morph provider for sync support)
VM_ID=$(devsh start -p morph --json | python3 -c "import json,sys; print(json.load(sys.stdin)['id'])")

# Sync backup to VM
devsh sync "$VM_ID" ./backup-staging/

# Run test harness
devsh exec "$VM_ID" "bash /tmp/test-restore.sh --manifest /tmp/manifest.json"

# Clean up
devsh delete "$VM_ID"

Note: devsh pve-lxc provider does NOT support devsh sync or direct SSH file transfer. Use morph provider for restore testing. For pve-lxc, use HTTP serve (python3 -m http.server + curl) as a workaround. ^[projects/agent-skills/compound/devsh-restore-testing.md]

Restore Target Pre-Inspection

Before restoring to any target host, run pre-inspection to verify readiness: ^[queries/hermes-backup-validation-restore-preinspection.md]

# Architecture
ssh <host> "uname -m"       # Expect: aarch64 or x86_64

# OS compatibility (critical for apt restore)
ssh <host> "cat /etc/os-release"

# Python version (Hermes requires 3.10+)
ssh <host> "python3 --version"

# Disk space (2GB+ recommended)
ssh <host> "df -h ~"

# Hermes already installed?
ssh <host> "hermes --version 2>/dev/null || echo NOT_INSTALLED"

# SSH key auth confirmed
ssh -o BatchMode=yes <host> "hostname"

devsh pve-lxc specific notes

• devsh sync NOT supported — use HTTP serve or morph provider
• systemctl --user FAILS (no user bus in LXC) — run gateway as system service or background process
• devsh exec works for all commands
• HTTP file transfer works (same 10.10.x.x/16 subnet)

File transfer methods (pve-lxc)

For backup archives larger than 32 KB (Caddy config, SSL certs, Hermes zip), use chunked base64, HTTP serve, or rsync bridge. The devsh_transfer helper in host-restore-cli.sh splits files into 30KB base64 chunks and reassembles on the remote side.

Lessons Learned

1. Bash JSON is fragile — Use python3 for final JSON assembly in discover.sh
2. systemctl is-active quirks — Prints to stdout even with stderr redirected; use &>/dev/null
3. hermes backup --tier is invalid — Hermes backup uses --quick for minimal, no flag for full
4. Restore across distros — Restoring apt sources across different distros (Debian→Ubuntu) breaks apt
5. SQLite WAL mode — Naively copying a SQLite DB with active WAL mode misses -wal/-shm files; use hermes backup or .backup command
6. Stop gateway before import — Always stop hermes-gateway.service before hermes import to avoid file lock conflicts ^[entities/hermes-backup-restore-guide.md]
7. sg01 baseline reference — sg01 provides known-good sizes: state.db ~427 MB, skills ~115 MB, HERMES_HOME ~2.2 GB ^[queries/hermes-backup-validation-restore-preinspection.md]
8. Model pinning for cost control — Pin mechanical SSH tasks to sonnet via backup-worker agent; keep orchestration in main session for user interaction quality
9. portfolio-lab is exclusive — The portfolio-lab profile selects ONLY portfolio_lab; any mix with generic groups, --all, or --restore-identity fails closed
10. Follow the authoritative recovery CLI contract — never invent a backup subcommand or --no-activate/--activate on portfolio_lab_recovery.py; use canonical verify --archive (backup) and canonical restore --archive --app-dir --web-root --target-mode dev|prod flags (restore)
11. Portfolio Lab backup is retrieval, not discovery or creation — explicit --source-archive + .tar.sha256 sidecar and independently trusted --trusted-sha256 into an explicit absolute --dest with resumable rsync (--partial-dir), then canonical verify; creation stays on the source host with canonical create (PORTFOLIO_LAB_PROJECT_DIR=<explicit repo> selects the python_runtime.sh project directory); no attestation claim and no cloud transfer from the retrieval path
12. Fresh-target restore executes a packaged bootstrap only after trust and safety gates — require the independently trusted --trusted-sha256, compare it and the sidecar against archive bytes on the target, validate every member (no traversal/absolute/symlink/device paths; exact recovery-manifest.json + tools/portfolio_lab_recovery.py), extract ONLY the exact bootstrap to controlled staging (never full-extract), then run canonical verify --archive on the target BEFORE canonical restore; production activation is never automatic — only the manual activate-prod command is printed

Related

• [[hermes-cli]] — Hermes CLI command reference
• [[entities/hermes-backup-restore-guide]] — Full backup/restore guide
• [[entities/sg01-host-infrastructure]] — Host-level infrastructure (Caddy, services, credstore)
• [[concepts/claude-code-agent-model-specification]] — Agent model pinning reference