homelab

Imported from zakaria1193/homelab (AGENTS.md). Install upstream with npx skills add zakaria1193/homelab. Copyright stays with the author.

Homelab Service Setup & Deployment Guidelines for Agents

This document defines mandatory guidelines and standards for creating, managing, and maintaining services within this homelab repository.

1. Self-Contained & Reproducible Service Architecture

[!IMPORTANT] Native CLI & Systemd Deployment Standard:

• Services MUST be deployed natively via local package managers (npx, npm, uv, or pip) and managed using system-level systemd daemons.
• DO NOT USE DOCKER OR DOCKER COMPOSE for service setups unless the user explicitly requests a containerized deployment.
• All service execution scripts, flags, and binaries must be managed within standard Makefile targets.

[!IMPORTANT] Full Visual Web UI Deployment Standard:

• When deploying services that provide a web interface (such as OpenHands), agents MUST verify whether the service package requires a full web stack launcher (e.g., npx @openhands/agent-canvas) versus a headless API-only server (e.g., agent-server).
• Service Makefile targets MUST launch the complete visual frontend stack by default so accessing http://<host>:<port>/ in a web browser renders the visual application interface rather than a raw JSON API status payload.

[!TIP] Systemd User Service Fallback: Service Makefile targets MUST check if non-interactive sudo is available (sudo -n true). If passwordless sudo is unavailable, Makefiles MUST automatically fall back to user-level systemd daemons (systemctl --user with unit files stored under ~/.config/systemd/user/) to ensure automated single-command setup without hanging on interactive password prompts.

Every service directory under services/ (e.g., services/AI/arrMcpAI, services/AI/hermesAI, services/AI/paperclipAI) MUST be 100% self-contained and reproducible on a fresh machine.

Required Files in Every Service Directory:

• Makefile: Standard automation script for installation, setup, systemd management, and logging.
• .env.example: Clean environment configuration template with default variables and comments for API keys, network hosts, and secrets.
• .env: Local runtime environment file (git-ignored, created via make env-setup).
• <service-name>.service.template: Systemd unit template for system-level daemon deployment.
• README.md: Complete documentation outlining overview, directory structure, quick-start guide, available make targets, and customization options.

2. Standard Makefile Targets & Requirements

Every service Makefile MUST implement the following standardized target interface to guarantee single-command reproducibility:

3. Cloudflare Tunneling Compatibility (cloudflared)

Services deployed in this homelab are accessed remotely and across the local network via Cloudflare Tunnels (cloudflared), which may run on the same local host OR on another machine on the LAN (e.g., Raspberry Pi 4 gateway, Dell primary server).

To ensure 100% compatibility with Cloudflare Tunnels:

1. Network Binding:
   • Web dashboards and API gateways MUST bind to 0.0.0.0 or be configurable via environment variables (HOST=0.0.0.0 or DASHBOARD_HOST=0.0.0.0).
2. Port Allocation & Conflict Prevention:
   • Agents MUST inspect active listening ports on the host (ss -tuln) and search existing service configs BEFORE choosing a default service port to avoid collisions with active homelab services (e.g., karakeep on port 3000).
3. Allowed Hostnames & CORS:
   • Services must allow reverse-proxy hostnames and IP addresses configured via .env or Makefile options (e.g., HOSTS="192.168.1.10 my-service.domain.com").
4. Authentication & Security:
   • Exposed web interfaces (such as dashboards) MUST support basic authentication or token authentication (HERMES_DASHBOARD_BASIC_AUTH_...) when accessible via Cloudflare Tunnels.

4. Environment Secrets & Git Hygiene

• NEVER commit a plaintext .env or API key into git. This repository is public: anything pushed unencrypted is public forever, even if deleted in a later commit.
• Always update .env.example whenever new environment variables or feature flags are added.

[!IMPORTANT] .env files ARE committed — encrypted with git-crypt. A service is only reproducible on a fresh machine if its secrets travel with it, so each service's .env is committed as ciphertext rather than ignored.

Committing a service .env via git-crypt

Run these from the repository root, in order:

1. Register the file, one path per line — append to /.gitattributes:

   services/<path>/.env filter=git-crypt diff=git-crypt
   

   Never broaden this to a bare *.env glob: that would sweep every other service's uncommitted secrets into the public history.
2. Un-ignore it in the service's own .gitignore, since the root .gitignore excludes .env globally:

   # Tracked, but git-crypt-encrypted (see /.gitattributes)
   !.env
   

3. Confirm the repo is unlocked (the filter is a no-op when locked, and the file would be committed as plaintext):

   git-crypt status -e          # must list the new path as "encrypted"

4. Stage it — -f is required because of the root .gitignore:

   git add -f services/<path>/.env

5. Verify the staged blob is ciphertext BEFORE committing. It must begin with the \0GITCRYPT\0 magic header:

   git show :services/<path>/.env | head -c 12 | xxd
   # 00000000: 0047 4954 4352 5950 5400 ....    .GITCRYPT..

   Grep the staged blob for a known secret value as a second check. If either test shows plaintext, STOP and do not commit.
6. Commit and push as usual.

New machines run git-crypt unlock (or git-crypt unlock <keyfile>) once; collaborators are added with git-crypt add-gpg-user <key-id>.

5. Mandatory Service Decommissioning & Removal Protocol

When removing or decommissioning a service from this repository, agents MUST strictly follow this 4-step sequence:

1. Stop & Disable Running Systemd Service: Run make systemd-stop or make clean within the target service directory (sudo systemctl stop <service> and sudo systemctl disable <service>).
2. Clean Up Systemd Unit Files: Remove /etc/systemd/system/<service>.service and run sudo systemctl daemon-reload to unregister the unit from systemd completely.
3. Clean Up Runtime & Local Files: Remove local runtime sockets, caches, or state directories if necessary.
4. Git Removal & Commit: Remove the service directory from git (git rm -r services/.../<service-name>) and commit the removal with a clean, descriptive commit message (e.g. feat(services): remove deprecated <service-name> service).

6. Registering a Service on the Cockpit

services/status is the homelab's operating console, not a maintenance report: each group shows an always-visible row of the things you click, and folds state, logs and shells away behind logs & shells. Every new or changed service MUST be reflected in services/status/services.conf, then picked up with make -C services/status upgrade.

1. Add a section named exactly as the card should read. Config order decides the order of the groups and of the entries inside one.

2. Give it both URLs. link is the LAN address, remote is the Cloudflare hostname. The page leads with whichever matches how the browser reached it and keeps the other one click away, so a service published through the tunnel MUST carry both. Keep remote in sync with the published-routes table in services/status/README.md whenever a tunnel route changes.

3. Point dir at the service's Makefile. That is where the browser shell opens, and the session starts on make help. For type = docker, dir is the directory holding docker-compose.yml: the card then offers shell (inside the container) and compose (on the host, starting on docker compose ps).

4. Add pinned = 1 only for services you must reach even when they are down. Everything else appears in the quick row automatically while it is up, and is otherwise reachable in the folded block.

5. Use type = shell for a launcher. It is a terminal with no service behind it — never probed, never counted in the totals — and command is typed into a login shell, so your own .zshrc aliases work verbatim:

   [home-assistant (claude)]
   group   = Home
   type    = shell
   command = make claude
   dir     = services/AI/hass-sshfs-bots

6. One entry per systemd unit. A service that runs several units — such as claude-rc-ai, which needs one Remote Control process per workspace — gets one section per unit, each pinning its own unit = .... Remote Control instances are the exception to writing this by hand: create them from the cockpit's Claude sessions page (/claude-rc) and it writes the env files, starts the unit and appends the section for you.

7. Name the entry after the thing. The logo on its chip is looked up from that name ([jellyfin], [docker], [home-assistant] need no icon key), and an entry pointing at %(pi)s is badged as running on the Raspberry Pi automatically. headline = 1 lifts an entry into the page header next to the totals; it is for what you operate the homelab with, so keep it to one or two.

8. Decommissioning (§5) MUST also delete the service's services.conf section in the same commit, so the cockpit never shows a unit that no longer exists.

7. Mandatory Slack Alerting & Webhook Routing Standard

[!IMPORTANT] Automatic Notification Integration: Whenever creating, updating, or configuring a service that supports notifications or webhooks (e.g. Radarr, Sonarr, Readarr, Jellyfin, Prowlarr, monitoring daemons, backup cron jobs), agents MUST automatically configure it with the homelab Slack incoming webhook infrastructure.

Single-Channel Webhook Constraint & Routing Architecture:

• In Slack, an Incoming Webhook URL is tied to a single specific channel chosen at creation time.
• To route notifications cleanly across channels (e.g. #general, #media-server, #alerts), homelab maintains a central webhook routing registry:
  • Runtime config: ~/.config/homelab/slack_webhooks.csv (or .env / tools/slack_webhooks.example.csv)
  • Format:

    channel,webhook_url,description
    #general,https://hooks.slack.com/services/...,General homelab notifications & health alerts
    #media-server,https://hooks.slack.com/services/...,Media lifecycle events (Sonarr, Radarr, Readarr, Jellyfin)

Standards for Services:

1. Media Stack Services (services/media):
   • MUST route to the #media-server webhook URL if defined in ~/.config/homelab/slack_webhooks.csv, falling back to SLACK_WEBHOOK_URL in ~/.config/homelab/slack.env.
   • Radarr, Sonarr, Readarr, Prowlarr: Configure via REST API (/api/v3/notification or /api/v1/notification).
   • Jellyfin: Configure via Jellyfin.Plugin.Webhook.xml.
2. Maintenance, Cron Jobs & Upgrades (services/AI/Makefile, backup tasks):
   • MUST invoke tools/slackbot-notify.sh with appropriate status flags (--status ok|warn|error) and titles.
   • On upgrade failure, services MUST invoke tools/auto-heal-and-notify.sh so agy attempts autonomous healing before alerting the operator on Slack.

Home Assistant Management Rules

• NEVER perform a full host/Pi reboot (sudo reboot) for Home Assistant changes.
• Always only reload the specific component or restart HA core. Never reboot the underlying Pi host.