Imported from fred-agent/fred-deployment-factory (.claude/skills/fredlab-observability/SKILL.md). Install upstream with npx skills add fred-agent/fred-deployment-factory --skill fredlab-observability. Copyright stays with the author.

Fredlab Observability

A read-only investigative session against a live Fred deployment (today: fredlab on GKE, project fredlab-playground, namespace fred-demo) — you observe the cluster's own logs, metrics, and audit trail directly, form evidence-backed findings, and either hand the operator an exact command to fix what's safe to template or propose a docs/BACKLOG.md entry for what isn't. You never mutate the cluster yourself.

Relationship to fred's live-observability-session skill: that one drives a local dev stack by hand (developer clicks the UI, you tail stdout) and lives in the fred monorepo. This skill reads a live, unattended GKE cluster's own signal — no developer driving anything, no local dev stack — and lives in fred-deployment-factory because platform/infra concerns (GCP log sinks, GKE PodMonitoring, Helm chart ownership) are this repo's domain, not fred's. Complementary, not overlapping.

No C3 twin exists yet. The roadmap (this repo's CLAUDE.md) is explicit: harden the GKE/C1 reference first, generalize later. Every workstream below must phrase findings as true today at C1 plus what would/wouldn't still hold at C3 (RFC-0001 §6's three knobs: secrets source, network segmentation, hosting/sovereignty — nothing else changes by classification level) — never assume a classified twin's specifics, never claim something is "C3-compliant" when what you actually checked was C1.

Preconditions

• gcloud authenticated, project set to fredlab-playground; kubectl context pointed at fredlab-playground-gke (gcloud container clusters get-credentials ..., see docs/DEPLOY-CLOUD.md §1.1 if not already done).
• The live instance's namespace is fred-demo, not default. Several scripts (bin/fredlab-status.sh, bin/fredlab-gcp-log-retention.sh) default to default — always pass NAMESPACE=fred-demo explicitly, and say so if you catch yourself about to run one without it.
• Strict read-only stance, no exceptions: you may run gcloud logging read, gcloud logging sinks/buckets describe, any kubectl get|describe|logs|top, kubectl exec -- psql -c "SELECT ..." / curl against in-cluster services, and existing bin/fredlab-*.sh scripts in their default preview mode only. You never run --apply, kubectl apply/edit/delete/annotate, helm upgrade, argocd app sync, or any write query. This matches this repo's CLAUDE.md rule ("platform mutations go through bin/fredlab-*.sh, one step at a time, never ad-hoc kubectl/helm/argocd") applied to its logical conclusion for an investigative session: you hand the operator the exact command; he runs it.
• Re-run bin/fredlab-status.sh (with the namespace above) once at the start of any session as a baseline sanity check before doing anything workstream-specific — don't investigate a "problem" that's actually just a mid-rollout pod restart.

Reaching each stream

The four workstreams

Run whichever subset the argument-hint (or the operator's actual question) calls for; default to all four for a general "is fredlab healthy" ask. Each workstream's detailed procedure, what "good" looks like, and the specific known gaps to check for live in its own reference file — read the relevant one before starting, don't rely on memory of a prior session (facts here rot fast: a gap found today may be fixed next week, and a new one may appear).

1. Scale/capacity readiness — references/scale-readiness.md. What "would this hold up at ~1000 concurrent users" actually means to check today (headroom, connection pools, queue depth, autoscaling behavior), including the direct-query workarounds needed because several relevant metrics don't reach Prometheus yet. Explicitly a reconnaissance pass on the C1 playground, not empirical proof — real load-test validation belongs on the future C3 twin.
2. GDPR / data-hygiene audit — references/gdpr-data-hygiene.md. The closed checklist: is the log-retention exclusion active and in the right namespace, is OpenSearch's auth gap still open, any new PII-in-logs regression, has log_level drifted off its safe INFO default anywhere, is the security audit trail's downstream durability decided (today: no, anywhere).
3. KPI/Prometheus health and innovation — references/kpi-health-and-gaps.md. Confirm existing dashboards/scrape targets are actually healthy, diff "who's scraped" against "who exposes /metrics", and — the "innovative" priority — always close this workstream by proposing at least one concrete, buildable new KPI or panel.
4. Remediation handoff — references/remediation-snippets.md. For every finding across the three workstreams above, decide: is this safe to hand over as an exact ready-to-run command (config one-liners, namespace fixes, a new scrape target), or is it a real security/architecture decision that only gets a docs/BACKLOG.md entry proposal? Never blur the two.

Reporting shape

Every finding needs all four parts, same discipline as live-observability-session's protocol, adapted for an unattended cluster (no "developer just did X" step to anchor on — anchor on the query instead):

• Evidence — the actual command you ran and its real output, quoted, not paraphrased.
• Channel — which of stdout / Cloud Logging / OpenSearch / Prometheus-Grafana / audit trail.
• Classification — confirmed regression, config gap, already-tracked (cite the existing docs/BACKLOG.md ID if one exists), or new improvement opportunity.
• Next step — one of: "ready-to-run command below" (§ remediation-snippets.md pattern), "propose a new docs/BACKLOG.md entry" (state the theme + next available ID — re-grep the file for the actual current max, IDs listed anywhere in this skill's reference files rot within weeks), or "needs the operator's judgment call, here are the options."

A finding missing any of these four isn't ready to report yet — don't report "OpenSearch has no auth" without also stating what stream it affects, whether it's new since the last check, and what the concrete next step is.

Ending the session

Nothing to stop (no background processes started, unlike live-observability-session — this skill never launches long-running local backends). Close out by: listing every command you handed the operator that's still pending his approval/run, and confirming which workstreams you actually ran vs. skipped (so a "quick GDPR check" doesn't get silently reported as if it were a full audit).