Imported from ByronWilliamsCPA/plugin (plugins/wff-chat/skills/observability-and-instrumentation/SKILL.md). Install upstream with npx skills add ByronWilliamsCPA/plugin --skill observability-and-instrumentation. Copyright stays with the author.

Observability and Instrumentation

Ported skill. Adapted from addyosmani/agent-skills (MIT License), commit a5f0b17, retrieved 2026-06-18. Adapted to our standards: em-dashes removed, cross-references repointed to our skills and agents, RAD markers added. Code examples are illustrative (TypeScript); the principles are language-agnostic and apply equally to our Python services (use structlog / logging for structured logs, prometheus_client or OpenTelemetry for metrics).

Overview

Code you cannot observe is code you cannot operate. Observability is the ability to answer "what is the system doing and why?" from the outside, using the telemetry the code emits. Instrumentation is not a post-launch add-on; it is written alongside the feature, the same way tests are. If a feature ships without telemetry, the first user-reported bug becomes archaeology instead of a query.

When to Use

• Building any feature that will run in production
• Adding a new service, endpoint, background job, or external integration
• A production incident took too long to diagnose ("we could not tell what happened")
• Setting up or reviewing alerting rules
• Reviewing a PR that adds I/O, retries, queues, or cross-service calls

NOT for:

• Diagnosing a failure happening right now: use the debug-tests and systematic-debugging skills (observability is what makes those fast next time)
• Profiling and optimizing measured slowness: use the performance-optimization skill
• Launch-day monitoring checklists and rollback triggers: see the shipping-and-launch skill; this skill covers the instrumentation that feeds them

Process

1. Define "working" before instrumenting

Telemetry without a question is noise. Before adding any instrumentation, write down 2 to 4 questions an on-call engineer will ask about this feature:

FEATURE: checkout payment retry
QUESTIONS ON-CALL WILL ASK:
1. What fraction of payments succeed on first attempt vs after retry?
2. When a payment fails permanently, why? (provider error? timeout? validation?)
3. Is the payment provider slower than usual?
-> Every signal below must help answer one of these.

If you cannot name the questions, you are not ready to instrument: you will log everything and learn nothing.

2. Pick the right signal for each question

Rule of thumb: metrics tell you that something is wrong, traces tell you where, logs tell you why.

3. Structured logging

Log events, not prose. Every log line is a JSON object with a stable event name and machine-readable fields:

// BAD: string interpolation, unqueryable, inconsistent
logger.info(`Payment ${id} failed for user ${userId} after ${n} retries`);

// GOOD: stable event name plus structured fields
logger.warn({
  event: 'payment_failed',
  paymentId: id,
  provider: 'stripe',
  errorCode: err.code,
  attempt: n,
}, 'payment failed');

Log levels, used consistently:

Correlation IDs are mandatory. Generate (or accept) a request ID at the system boundary and attach it to every log line, span, and outbound call. Without it, you cannot reconstruct a single request from interleaved logs:

// Express: child logger per request, ID propagated downstream
app.use((req, res, next) => {
  req.id = req.headers['x-request-id'] ?? crypto.randomUUID();
  req.log = logger.child({ requestId: req.id });
  res.setHeader('x-request-id', req.id);
  next();
});

Never log secrets, tokens, passwords, or full PII. This is a hard rule (see our security skill and the owasp-* agents): telemetry pipelines are a classic data-leak path. Allowlist fields; do not log whole request bodies.

# RAD: telemetry is a security-sensitive sink.
# #CRITICAL: log fields are allowlisted; no secrets/PII reach the log pipeline.
# #VERIFY: grep emitted staging logs for token/password/email before sign-off.

4. Metrics

For request-driven services, instrument RED on every endpoint and every external dependency: Rate (requests/sec), Errors (failure rate), Duration (latency histogram, not average). For resources (queues, pools, hosts), use USE: Utilization, Saturation, Errors.

The vendor-neutral path is the OpenTelemetry metrics API (same SDK and context as step 5). The example below uses Prometheus' prom-client, one common backend choice, not the only one; the RED/USE and cardinality rules are identical either way.

import { Histogram } from 'prom-client';

const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route', 'status_class'],  // '2xx', not '200'
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});

Cardinality is the failure mode. Every unique label combination is a separate time series. Labels must come from small, fixed sets (route template, status class, provider name). Never use user IDs, raw URLs, error messages, or other unbounded values as labels: that belongs in logs and traces.

OK as label:    route="/api/tasks/:id"   status_class="5xx"   provider="stripe"
NEVER a label:  user_id, email, request_id, full URL, error message text

Track averages never, percentiles always: an average hides the 1% of users having a terrible time. Use histograms and read p50/p95/p99.

5. Distributed tracing

Use OpenTelemetry; it is the vendor-neutral standard, and auto-instrumentation covers HTTP, gRPC, and common DB clients with near-zero code:

// tracing.ts, must be imported before anything else
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const sdk = new NodeSDK({
  serviceName: 'checkout-service',
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

Add manual spans only around meaningful internal units of work (for example applyDiscounts, chargeProvider) and attach the attributes on-call will filter by. Propagate context across every async boundary (HTTP headers, queue message metadata) or the trace dies at the gap. Sample head-based at a low rate by default; keep 100% of errors if your backend supports tail sampling.

6. Alerting

Alert on symptoms users feel, not on causes:

SYMPTOM (page-worthy):           CAUSE (dashboard, not a page):
error rate > 1% for 5 min        CPU at 85%
p99 latency > 2s                 one pod restarted
queue age > 10 min               disk at 70%

Cause-based alerts fire when nothing is wrong and miss failures you did not predict. Symptom-based alerts fire exactly when users are hurt, regardless of the cause.

Rules for every alert you create:

1. It must be actionable. If the response is "ignore it, it self-heals", delete the alert.
2. It links to a runbook, even three lines: what it means, first query to run, escalation path.
3. It has a threshold and duration justified by the SLO or by historical data, not by a guess.
4. Use two severities only: page (user-facing, act now) and ticket (degradation, act this week). A third tier becomes noise that trains people to ignore everything.

Security alerting is a distinct axis from symptom alerting. The symptom rules above optimize for "users are hurt", which is the right default for reliability and the wrong one for an attack in progress: credential stuffing, enumeration, and scripted abuse degrade nothing a user feels until they succeed. Add rules for authentication-failure and authorization-denial spikes alongside the symptom set. Standards manifest OPS-006 covers this and requires three things a reliability alert usually leaves implicit: the rule is committed (not configured only in a vendor console), it names its destination channel, and it carries a recorded test-fire timestamp. An alert that pages nobody is not alerting.

Related manifest checks in the operations domain (docs/standards-manifest.yaml): OPS-005 (security events emitted against a documented taxonomy, and greppable in source, so a taxonomy cannot document events the code never emits) and OPS-004 (log secret redaction, proven by a test rather than by the redaction helper's presence).

This skill is the human-facing narrative for instrumenting one service. The OPS-* checks are the durable half: they land in the compliance master log, get delta caching, fleet escalation at the 3-repo threshold, and staleness detection, none of which a skill provides. Use both.

7. Verify the telemetry itself

Instrumentation is code; it can be wrong. Before calling the work done, trigger the paths and look at the actual output:

• Force an error in staging, find it in the logs by requestId, confirm fields are structured (not [object Object])
• Send test traffic, confirm metric series appear with the expected labels and sane values
• Follow one request across services in the tracing UI, no broken spans
• Fire each new alert once (lower the threshold temporarily), confirm it reaches the right channel and the runbook link works

Common Rationalizations

Red Flags

• A feature PR with retries, queues, or external calls and zero new telemetry
• Log lines built by string interpolation instead of structured fields
• No correlation/request ID, each log line is an orphan
• Metrics labeled with user IDs, raw URLs, or error message text (cardinality bomb)
• Latency tracked as an average with no percentiles
• Alerts that fire daily and get acknowledged without action
• Alerts on causes (CPU, memory) paging humans while user-facing error rate is unmonitored
• Secrets, tokens, or full request bodies appearing in logs
• "It works on my machine" as the only evidence a production feature is healthy

Verification

After instrumenting a feature, confirm:

• The on-call questions for this feature are written down, and each signal maps to one
• All log output is structured (JSON), with stable event names and a correlation ID on every line
• No secrets, tokens, or unredacted PII in any log line (spot-check actual output)
• RED metrics exist for every new endpoint and every external dependency, with bounded label sets
• Latency is a histogram; p95/p99 are queryable
• A single request can be followed end-to-end in the tracing UI without broken spans
• Every new alert is symptom-based, has a runbook link, and was test-fired once
• An induced failure in staging was located via telemetry alone, without reading the source