Imported from Way-Of/wayofmono (packages/@wayofmono/wo-user-extra/skills/otel_instrumentation/SKILL.md). Install upstream with npx skills add Way-Of/wayofmono --skill otel_instrumentation. Copyright stays with the author.

OpenTelemetry Application Instrumentation

You are an expert OpenTelemetry instrumentation engineer. Produce prescriptive, opinionated guidance — not tutorials. Every recommendation must be production-grade.

Entrypoint: Detect → Scope → Act

1. Detect Language/Framework

2. Determine Scope

1. Resource Attributes (CRITICAL)

Resource attributes identify your service. This is the single highest-impact configuration for observability.

Required

Environment Variables

OTEL_SERVICE_NAME=order-service
OTEL_RESOURCE_ATTRIBUTES=service.namespace=commerce,deployment.environment.name=production
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Kubernetes: Use Downward API

env:
  - name: OTEL_SERVICE_NAME
    value: "order-service"
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
  - name: POD_NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
  - name: NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: >-
      k8s.pod.name=$(POD_NAME),
      k8s.namespace.name=$(POD_NAMESPACE),
      k8s.node.name=$(NODE_NAME),
      service.namespace=commerce,
      deployment.environment.name=production

Set attributes at SDK level — do NOT rely solely on the Collector's k8sattributes processor.

2. Traces / Spans

Naming: {VERB} {OBJECT}, Low Cardinality

Never include IDs, query params, or request bodies in span names.

Span Kind

Status Code Rules

Exception Recording

Record exceptions in logs with trace context, not span events. The Span Event API is being phased out.

// BAD: span.recordException(error);
// GOOD: Log with trace context
logger.error('Payment failed', {
  error: error.message,
  'error.stack': error.stack,
  'order.id': orderId,
});

Span Hygiene

1. No CLIENT root spans — wrap outgoing calls in a SERVER or INTERNAL parent
2. No orphan spans — propagate context across async boundaries
3. Limit INTERNAL spans to ~10/trace — only business-significant operations
4. Limit sub-1ms spans to ~20/trace — rarely add diagnostic value
5. Always use AlwaysOn sampler at SDK — defer sampling to the Collector (see otel_collector)

3. Metrics

Instrument Selection

Naming

1. Check otel_semantic_conventions first
2. Dot-separated namespaces: http.server.request.duration
3. Never include unit in name (duration not duration_seconds)
4. UCUM units: s, ms, By, {request}

Cardinality Management (CRITICAL)

Every unique attribute combination = a new time series. This is the #1 cost driver.

Hard rules:

• Never use user IDs, request IDs, or trace IDs as metric attributes
• Never use unbounded strings (URLs, error messages) as metric attributes
• Limit enum attrs to <20 values; use _OTHER for long tail
• Use url.template not url.path on metrics

RED Pattern

One Histogram with these attributes covers Rate, Errors, and Duration:

• http.request.method, http.response.status_code, url.template, error.type

4. Structured Logging

Rules

1. Structured key-value pairs only — never string interpolation
2. Inject trace context — trace_id + span_id in every log line
3. stdout only — let infrastructure route logs
4. Single-line JSON for machine parsing

Trace Correlation

// Node.js (pino)
const logger = pino({
  mixin() {
    const span = trace.getActiveSpan();
    if (!span) return {};
    const ctx = span.spanContext();
    return { trace_id: ctx.traceId, span_id: ctx.spanId, trace_flags: ctx.traceFlags };
  },
});

// Go (slog)
func LogWithTrace(ctx context.Context, logger *slog.Logger, msg string, attrs ...slog.Attr) {
    span := trace.SpanFromContext(ctx)
    if span.SpanContext().IsValid() {
        attrs = append(attrs,
            slog.String("trace_id", span.SpanContext().TraceID().String()),
            slog.String("span_id", span.SpanContext().SpanID().String()),
        )
    }
    logger.LogAttrs(ctx, slog.LevelInfo, msg, attrs...)
}

# Python (structlog)
def add_trace_context(logger, method_name, event_dict):
    span = trace.get_current_span()
    if span.get_span_context().is_valid:
        ctx = span.get_span_context()
        event_dict["trace_id"] = format(ctx.trace_id, "032x")
        event_dict["span_id"] = format(ctx.span_id, "016x")
    return event_dict

Severity Levels

Stack Traces

Always single-line JSON — never multi-line:

logger.error('Operation failed', {
  'exception.type': error.constructor.name,
  'exception.message': error.message,
  'exception.stacktrace': error.stack.replace(/\n/g, '\\n'),
});

5. Sensitive Data Prevention (CRITICAL)

Never-Instrument List

Sanitization Rules

• URLs: Strip query params. Use url.path + url.scheme + server.address, not http.url
• DB statements: Parameterized form only: SELECT * FROM users WHERE id = ?
• Headers: Never capture authorization, cookie, set-cookie

SpanProcessor Redaction (Safety Net)

class RedactingSpanProcessor {
  onEnd(span) {
    const sensitive = ['password', 'token', 'secret', 'authorization', 'cookie', 'credit_card'];
    for (const [key, value] of Object.entries(span.attributes)) {
      if (sensitive.some(k => key.toLowerCase().includes(k))) {
        span.attributes[key] = '[REDACTED]';
      }
    }
  }
  onStart() {}
  shutdown() { return Promise.resolve(); }
  forceFlush() { return Promise.resolve(); }
}

For Collector-side redaction as defence-in-depth, see otel_ottl.

6. Language Setup Guides

Node.js

npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-grpc @opentelemetry/exporter-metrics-otlp-grpc \
  @opentelemetry/exporter-logs-otlp-grpc @opentelemetry/resources @opentelemetry/semantic-conventions

// instrumentation.ts — load BEFORE application code via --require or --import
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc';
import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-grpc';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { BatchLogRecordProcessor } from '@opentelemetry/sdk-logs';
import { Resource } from '@opentelemetry/resources';
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';
import { randomUUID } from 'crypto';

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: process.env.OTEL_SERVICE_NAME || 'my-service',
    [ATTR_SERVICE_VERSION]: process.env.SERVICE_VERSION || 'unknown',
    'service.instance.id': randomUUID(),
    'deployment.environment.name': process.env.NODE_ENV || 'development',
  }),
  traceExporter: new OTLPTraceExporter(),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter(),
    exportIntervalMillis: 60_000,
  }),
  logRecordProcessor: new BatchLogRecordProcessor(new OTLPLogExporter()),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();
process.on('SIGTERM', () => sdk.shutdown().then(() => process.exit(0)).catch(() => process.exit(1)));

node --require ./instrumentation.ts src/index.ts
# ESM: node --import ./instrumentation.ts src/index.ts

Custom spans:

import { trace, SpanStatusCode } from '@opentelemetry/api';
const tracer = trace.getTracer('my-service');

async function processOrder(orderId: string) {
  return tracer.startActiveSpan('process order', async (span) => {
    try {
      span.setAttribute('order.id', orderId);
      return await doWork();
    } catch (error) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
      throw error;
    } finally {
      span.end();
    }
  });
}

Custom metrics:

import { metrics } from '@opentelemetry/api';
const meter = metrics.getMeter('my-service');

const requestDuration = meter.createHistogram('http.server.request.duration', {
  description: 'Duration of HTTP server requests',
  unit: 's',
});

Go

go get go.opentelemetry.io/otel go.opentelemetry.io/otel/sdk go.opentelemetry.io/otel/sdk/metric \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
  go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

func Setup(ctx context.Context) (func(context.Context) error, error) {
    res, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceName(os.Getenv("OTEL_SERVICE_NAME")),
            semconv.ServiceVersion(os.Getenv("SERVICE_VERSION")),
            semconv.DeploymentEnvironmentName(os.Getenv("ENVIRONMENT")),
            semconv.ServiceInstanceID(uuid.New().String()),
        ),
    )
    if err != nil { return nil, err }

    traceExp, _ := otlptracegrpc.New(ctx)
    tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(traceExp), sdktrace.WithResource(res))
    otel.SetTracerProvider(tp)

    metricExp, _ := otlpmetricgrpc.New(ctx)
    mp := metric.NewMeterProvider(
        metric.WithReader(metric.NewPeriodicReader(metricExp, metric.WithInterval(60*time.Second))),
        metric.WithResource(res),
    )
    otel.SetMeterProvider(mp)

    return func(ctx context.Context) error {
        if err := tp.Shutdown(ctx); err != nil { return err }
        return mp.Shutdown(ctx)
    }, nil
}

Custom spans:

tracer := otel.Tracer("my-service")

func ProcessOrder(ctx context.Context, orderID string) error {
    ctx, span := tracer.Start(ctx, "process order")
    defer span.End()
    span.SetAttributes(attribute.String("order.id", orderID))

    if err := doWork(ctx); err != nil {
        span.SetStatus(codes.Error, err.Error())
        return err
    }
    return nil
}

Python

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install  # Auto-detect instrumentations

resource = Resource.create({
    "service.name": os.getenv("OTEL_SERVICE_NAME", "my-service"),
    "service.version": os.getenv("SERVICE_VERSION", "unknown"),
    "service.instance.id": str(uuid.uuid4()),
    "deployment.environment.name": os.getenv("ENVIRONMENT", "development"),
})

tp = TracerProvider(resource=resource)
tp.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(tp)

reader = PeriodicExportingMetricReader(OTLPMetricExporter(), export_interval_millis=60000)
mp = MeterProvider(resource=resource, metric_readers=[reader])
metrics.set_meter_provider(mp)

Or zero-code: opentelemetry-instrument python app.py

Custom spans:

tracer = trace.get_tracer("my-service")

@tracer.start_as_current_span("process order")
def process_order(order_id: str):
    span = trace.get_current_span()
    span.set_attribute("order.id", order_id)

Java

Recommended: Java agent (zero-code):

java -javaagent:opentelemetry-javaagent.jar \
  -Dotel.service.name=my-service \
  -Dotel.exporter.otlp.endpoint=http://otel-collector:4317 \
  -jar my-app.jar

Custom spans:

Tracer tracer = GlobalOpenTelemetry.getTracer("my-service");
Span span = tracer.spanBuilder("process order").startSpan();
try (Scope scope = span.makeCurrent()) {
    span.setAttribute("order.id", orderId);
    doWork();
} catch (Exception e) {
    span.setStatus(StatusCode.ERROR, e.getMessage());
    throw e;
} finally {
    span.end();
}

.NET

dotnet add package OpenTelemetry.Extensions.Hosting OpenTelemetry.Instrumentation.AspNetCore OpenTelemetry.Exporter.OpenTelemetryProtocol

builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService(
        serviceName: builder.Configuration["OTEL_SERVICE_NAME"] ?? "my-service",
        serviceVersion: typeof(Program).Assembly.GetName().Version?.ToString()))
    .WithTracing(t => t.AddAspNetCoreInstrumentation().AddHttpClientInstrumentation().AddOtlpExporter())
    .WithMetrics(m => m.AddAspNetCoreInstrumentation().AddHttpClientInstrumentation().AddOtlpExporter());

Ruby

gem install opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-all

OpenTelemetry::SDK.configure do |c|
  c.service_name = ENV.fetch('OTEL_SERVICE_NAME', 'my-service')
  c.service_version = ENV.fetch('SERVICE_VERSION', 'unknown')
  c.use_all
end

7. Validation Checklist

Pre-Flight

env | grep OTEL_
curl -sf http://otel-collector:4317 || echo "Collector unreachable (gRPC)"
curl -sf http://otel-collector:4318/v1/traces || echo "Collector unreachable (HTTP)"
# Protocol must match: grpc→:4317, http/protobuf→:4318

Backend Checks

Signal Validation

Traces: correct kind, no orphans, ERROR only on failures, parameterized names Metrics: bounded cardinality (<1K series/metric), correct units, no duplicate names Logs: trace_id+span_id present, structured JSON, correct severity, no sensitive data

Cross-References

• Attribute naming → otel_semantic_conventions
• Collector config → otel_collector
• Collector-side transforms/redaction → otel_ottl