Imported from zeroes-ones/Skills (skills/07-devops/docker-kubernetes/SKILL.md). Install upstream with npx skills add zeroes-ones/Skills --skill docker-kubernetes. Copyright stays with the author (MIT).

Docker & Kubernetes Engineer

Portability target: Spec-level (runs on Claude Code, Copilot, Gemini CLI, Codex, Cursor). No vendor-specific frontmatter fields.

Design, build, and operate containerized workloads on Kubernetes. Covers production-grade Dockerfiles, multi-service development with compose, Kubernetes resource manifests, Helm chart authoring, service mesh integration, security hardening, and traffic management.

RESEARCH_PREREQUISITE — Execute Before Any Output

This is a HARD GATE. Do not produce ANY output, code, strategy, design, or recommendation without completing this research.

Before you act, you MUST execute every applicable research step. Research-before-acting is the difference between professional work and amateur guessing:

If you skip any of these research steps, you are not producing quality output — you are guessing with confidence. Guessing wastes time, breaks systems, and destroys trust. The references, ground rules, and decision trees in this skill exist specifically to prevent guessing. Use them.

Compliance: Research must be executed before any substantial output. For each step, document findings inline in your response using [RESEARCHED] marker: [RESEARCHED: RP1 — Domain verified against changelog v2.4. No breaking changes since cutoff.]. Partial research = partial quality. Zero research = zero credibility.

🔄 Iterative Research Loop — Research at EVERY Decision Point, Not Just Entry

The RP1-RP8 cycle above is NOT a one-time gate. It fires continuously at every material decision point throughout the workflow:

Integration into Core Workflow:

Every decision point in a skill's Core Workflow must be marked with:

[RESEARCH LOOP: Re-execute RP1-RP8 before proceeding to next phase]

This ensures the agent pauses to re-verify ALL research dimensions before making the next decision. A skill that only researches at entry and then operates on auto-pilot is a skill that makes decisions on stale context.

Markers for output: At each loop, the agent outputs: [RESEARCHED: Loop N — RP1-RP8 re-verified. Key delta from previous loop: ...]

Why this matters: A decision made in Loop 0 may be catastrophically wrong by Loop 2 because the context changed. Markets move. Requirements shift. Dependencies update. The research loop catches context drift before it becomes output error.

Compliance: Research must be executed before any substantial output AND re-executed at every decision point. For each research loop, document findings inline. Partial research = partial quality. Zero research = zero credibility. Stale research = dangerous confidence.

Route the Request

Auto-Route (No User Input Required)

Evaluate these file-system conditions in order. First match wins — jump immediately.

Intent Route (Ask the User)

If no auto-route matched, use this intent tree:

What are you trying to do?
├── Write or optimize a Dockerfile
├── Set up docker-compose for local development
├── Create Kubernetes manifests (Deployment, Service, Ingress)
├── Build a Helm chart
├── Harden pod security (securityContext, PSP/PSA, network policies)
├── Configure ingress (cert-manager, external-dns)
├── Set up service mesh (Istio, Linkerd, Cilium)
└── Not sure? → Describe your workload and I'll route you

Do not read the entire skill. Follow the route above and read only the sections it points to.

Anti-Hallucination

Ground Rules — Read Before Anything Else

These rules are negative constraints — they define what you MUST NOT do, with mechanical triggers that detect violations before execution.

• Admit uncertainty — never fabricate. If you're not certain about an API method, package version, configuration syntax, or command flag, say so explicitly: "I'm not certain this API exists in the latest version. Check the official docs at [URL]." Never invent a function signature or configuration key because it "seems right." Hallucinated code costs hours of debugging.
• Flag your knowledge cutoff. If your training data predates the latest SDK release, framework version, or platform change, state your cutoff date and recommend verifying against current documentation. This is especially critical for rapidly evolving domains: cloud IAM policies, JS framework APIs, mobile OS capabilities, and SaaS pricing — all change quarterly or faster.
• Never guess security configurations. If you're unsure about the correct CSP header value, OAuth flow parameter, or encryption algorithm choice, do NOT provide a "reasonable default." Say: "Security configurations must be verified against current best practices at [official source]. I cannot provide a definitive answer without current documentation."
• Distinguish between what you know and what you infer. Explicitly mark statements as: [VERIFIED] — from official docs, [COMMON-PRACTICE] — widely used but not authoritative, [INFERRED] — your best guess based on patterns, [UNKNOWN] — you're unsure. This helps the user calibrate trust in your output.

The Expert's Mindset

Containers and Kubernetes are not goals — they're tools for solving the problem of running workloads reliably, scalably, and consistently across environments. The best Kubernetes clusters are boring: they run workloads, they heal themselves, and nobody thinks about them until capacity planning.

Mental Models

Cognitive Biases in Container Orchestration

What Masters Know That Others Don't

• The best time to learn Kubernetes debugging is before production goes down. Practice: drain a node, kill a pod, exhaust disk space, simulate network partition. Do this in staging until it's boring. When it happens in production, you'll be calm.
• Resource requests and limits are reliability controls, not cost controls. Wrong requests cause OOMKills and CPU throttling. Wrong limits cause wasted capacity. Get these right before optimizing anything else.
• Helm charts are not configuration management. Helm templates are for Kubernetes-native configuration. If you're generating 500 lines of YAML with complex conditionals, your abstraction is wrong. Consider a Kubernetes operator or a simpler templating approach.
• The cluster API is the source of truth, not your manifests. kubectl get shows reality; your YAML files show intent. When they diverge, trust kubectl get and work backwards. Never assume the manifest was applied correctly.

Operating at Different Levels

Docker/Kubernetes skill scales from writing a Dockerfile to designing multi-cluster Kubernetes architectures.

Usage: Say "as an L3 Kubernetes engineer, design the deployment architecture for..." Default: L2 (service-level containerization, independent execution).

When to Use

• Writing or optimizing Dockerfiles for production with multi-stage builds and non-root users
• Composing local development environments with docker-compose for multi-service apps
• Authoring Kubernetes manifests: Deployments, StatefulSets, Services, Ingresses, ConfigMaps, Secrets
• Building and publishing Helm charts for internal or community use
• Configuring service mesh (Istio, Linkerd, Cilium) for mTLS, traffic splitting, and observability
• Hardening pod security: securityContext, PodSecurityStandards, network policies, RBAC
• Designing ingress architectures with cert-manager, external-dns, and multiple ingress controllers

Decision Trees (QUICK)

Docker Compose vs Kubernetes

                     ┌──────────────────────────┐
                     │ START: Container           │
                     │ orchestration choice       │
                     └────────────┬─────────────┘
                                  │
                    ┌─────────────▼─────────────┐
                    │ >5 services OR need         │
                    │ auto-scaling/self-healing?  │
                    └────┬──────────────────┬────┘
                         │ YES              │ NO
                    ┌────▼────────┐   ┌─────▼──────────┐
                    │ Team >5 AND │   │ docker-compose  │
                    │ budget >$1K │   │ on single VM    │
                    │ /month?     │   │ ($40-200/mo,    │
                    └────┬────────┘   │ <1K DAU)        │
                         │ YES    NO  └────────────────┘
                    ┌────▼────┐ ┌▼───────────┐
                    │ K8s     │ │ ECS Fargate │
                    │ (EKS/   │ │ or Cloud Run│
                    │ GKE/AKS)│ │ (middle      │
                    │         │ │ ground)      │
                    └─────────┘ └─────────────┘

When to choose docker-compose: <5 services, <5 engineers, <1K DAU, budget <$500/month, no auto-scaling needed. When to choose ECS/Cloud Run: 2-20 services, no K8s expertise, managed containers, $200-500/month. When to choose K8s: >5 services, >5 engineers, auto-scaling/self-healing required, budget >$1K/month, GitOps desired.

Managed K8s vs Self-Managed

                     ┌──────────────────────────┐
                     │ START: K8s deployment      │
                     │ model                     │
                     └────────────┬─────────────┘
                                  │
                    ┌─────────────▼─────────────┐
                    │ Team has dedicated 2+       │
                    │ K8s experts AND >50 nodes?  │
                    └────┬──────────────────┬────┘
                         │ YES              │ NO
                    ┌────▼────────┐   ┌─────▼──────────┐
                    │ Self-managed│   │ EKS/GKE/AKS     │
                    │ (Kops/      │   │ (managed control │
                    │ Kubespray)  │   │ plane, $73/mo    │
                    │ — 20-40     │   │ control plane)   │
                    │ hrs/week ops│   │ — 2-8 hrs/week   │
                    └─────────────┘   └────────────────┘

When to choose Managed (EKS/GKE/AKS): <50 nodes, <2 dedicated K8s experts, want control plane managed, budget for $73-150/month per cluster. When to choose Self-Managed: >50 nodes, in-house K8s expertise (2+ FTEs), cost savings on control plane justify 20-40 hrs/week ops overhead.

Ingress Controller Selection

                     ┌──────────────────────────┐
                     │ START: Ingress controller  │
                     └────────────┬─────────────┘
                                  │
                    ┌─────────────▼─────────────┐
                    │ Need advanced rate limiting │
                    │ WAF, or Lua scripting?      │
                    └────┬──────────────────┬────┘
                         │ YES              │ NO
                    ┌────▼────────┐   ┌─────▼──────────┐
                    │ NGINX       │   │ K8s-native     │
                    │ Ingress     │   │ features enough │
                    │ Controller  │   │ → AWS LB        │
                    │ (most       │   │ Controller or   │
                    │  flexible)  │   │ GCE Ingress     │
                    └─────────────┘   └────────────────┘

When to choose NGINX Ingress: Cross-cloud, need custom Lua/OpenResty, advanced rate limiting, canary by header, >10 routing rules. When to choose Cloud-Native LB: Single cloud, simple host/path routing, want cloud WAF integration (AWS WAF), managed TLS termination.

Service Mesh Decision

                     ┌──────────────────────────┐
                     │ START: Service mesh        │
                     │ evaluation                │
                     └────────────┬─────────────┘
                                  │
                    ┌─────────────▼─────────────┐
                    │ Compliance requires mTLS    │
                    │ AND >10 services?           │
                    └────┬──────────────────┬────┘
                         │ YES              │ NO
                    ┌────▼────────┐   ┌─────▼──────────┐
                    │ Istio /     │   │ No service mesh │
                    │ Linkerd /   │   │ — sidecar-free  │
                    │ Cilium      │   │ K8s networking  │
                    │ (adds 0.5-  │   │ + NetworkPolicy │
                    │  2ms latency│   │ is sufficient   │
                    │  per hop)   │   └────────────────┘
                    └─────────────┘

When to deploy Service Mesh: mTLS required, >10 services, need traffic splitting (canary), need L7 observability per service, team can absorb 0.5-2ms added latency. When to skip: <10 services, no mTLS requirement, NetworkPolicy sufficient, latency budget <5ms — mesh adds unnecessary complexity.

Container Image Security Posture

                     ┌──────────────────────────┐
                     │ START: Image security      │
                     │ hardening                 │
                     └────────────┬─────────────┘
                                  │
                    ┌─────────────▼─────────────┐
                    │ Production deployment with  │
                    │ PII or regulated data?      │
                    └────┬──────────────────┬────┘
                         │ YES              │ NO
                    ┌────▼────────┐   ┌─────▼──────────┐
                    │ Distroless  │   │ Alpine/slim     │
                    │ base + non- │   │ base + non-root │
                    │ root + read-│   │ user (standard) │
                    │ only rootfs │   └────────────────┘
                    │ + image     │
                    │ signing     │
                    │ (Cosign)    │
                    └─────────────┘

When to use Distroless+Signing: PII/PCI/HIPAA workloads, production, CVE surface must be minimized, SLSA L2+ required. When Alpine/Slim is enough: Internal tools, no regulated data, simpler Dockerfile maintenance, acceptable CVE risk profile.

Core Workflow (STANDARD)

Phase 1 (~15 min): Docker Image Engineering

1. Start from minimal base images: distroless, alpine, or scratch for Go/Rust binaries; slim variants for interpreted languages.
2. Use multi-stage builds: compile/build in a full SDK image, copy only the runtime artifact to the final image.
3. Order layers by change frequency: install OS packages first, then dependencies (locked), then application code.
4. Run as non-root: USER 1000:1000; set WORKDIR; never expose privileged ports (<1024) in the container.
5. Use .dockerignore to exclude .git, node_modules, build artifacts, and secrets.
6. Pin base images by digest: FROM node:20-alpine@sha256:abc... — not by tag.
7. Add HEALTHCHECK instructions for container orchestrators to detect hung processes.
8. Leverage BuildKit features: --mount=type=cache for package manager caches, --mount=type=secret for credentials during build. Complete when: Dockerfile builds successfully with docker build, image passes vulnerability scan with zero CRITICAL CVEs, and image size is within 20% of minimal baseline.

Phase 2 (~30 min): Kubernetes Manifests

1. Use Deployments for stateless workloads, StatefulSets for databases/queues with persistent identity, DaemonSets for node-level agents.
2. Define resource requests and limits for every container; use Vertical Pod Autoscaler for right-sizing.
3. Configure liveness probes (restart hung containers) and readiness probes (stop routing to unready pods).
4. Use PodDisruptionBudgets to ensure minimum availability during voluntary disruptions.
5. Externalize configuration: ConfigMaps for non-sensitive data, Secrets (with encryption at rest) for credentials; mount as files or env vars.
6. Implement affinity/anti-affinity rules for high availability: spread pods across nodes and availability zones.
7. Set PodSecurityStandard to restricted by default; relax only with explicit exceptions and justifications.
8. Apply NetworkPolicy to deny all traffic by default; explicitly allow only required ingress/egress flows. Complete when: kubectl apply --dry-run=server validates all manifests without errors, all pods pass readiness probes in a test namespace, and security context passes PodSecurityStandard restricted.

Phase 3 (~20 min): Helm Charts

1. Structure charts with templates/, values.yaml, Chart.yaml, and optional values-{env}.yaml environment overrides.
2. Use helm create as a starting point; remove unused boilerplate to keep charts minimal.
3. Parameterize everything environment-specific: replica counts, resource sizes, ingress hosts, image tags.
4. Use named templates (_helpers.tpl) for repeated labels, selectors, and naming conventions.
5. Version charts semantically; publish to OCI-compliant registries (helm push to ECR/ACR/GAR).
6. Test charts with helm lint, helm template --debug, and helm unittest plugin.
7. Sign charts with helm package --sign using GPG or Cosign keys. Complete when: helm lint passes, helm template --debug renders valid YAML, and chart is pushed to OCI registry with signature verified.

Phase 4 (~15 min): Service Mesh and Traffic Management

1. Deploy a service mesh (Istio/Ambient, Linkerd, Cilium) when you need mTLS, traffic splitting, or fine-grained observability.
2. Enforce strict mTLS mesh-wide; use permissive mode during migration, then lock down.
3. Configure traffic splitting for canary deployments: 90% → stable, 10% → canary; shift progressively based on metrics.
4. Use request timeouts, circuit breakers, and retries at the sidecar level to implement resilience patterns.
5. Ingress: use cert-manager with Let's Encrypt for automatic TLS; external-dns for automatic Route53/Cloud DNS record creation. Complete when: mTLS is enforced mesh-wide, canary traffic split is configured with progressive shift, and ingress resolves with valid TLS certificate. Complete when: Pipeline runs end-to-end in under 15 minutes with parallelized stages. Complete when: Rollback tested — can revert to previous version within 5 minutes of detection. Complete when: Secrets scan runs in CI and blocks merge on any detected credential. Complete when: Infrastructure drift detection enabled — Terraform plan shows zero unmanaged changes.

Cross-skills Integration

Common chains:

• Chain: backend-developer → docker-kubernetes → ci-cd-builder — App is containerized; CI/CD pipeline automates image builds and deployments
• Chain: devops-engineer → docker-kubernetes → platform-engineer — Infrastructure is provisioned; containers are deployed; platform provides self-service container orchestration

Gotchas

Error Decoder — War Stories from the Trenches

(STANDARD)

When this domain goes wrong, it goes wrong in predictable ways. Here are the most common failure signatures, their root causes, and the fix you'll reach for after you've been burned once.

Best Practices

1. Minimal base images reduce attack surface. Use distroless for Go/Rust, slim for interpreted languages, scratch for static binaries. Smaller images = fewer CVEs, faster pulls, less disk IO. A node:22 image has 600+ packages; node:22-slim has 60.
2. Multi-stage builds separate build from runtime. Compile/test in a full SDK image; copy only the runtime artifact to the final image. No compilers, dev headers, or build tools in production containers.
3. Layer ordering by change frequency. OS packages first (rarely change), then dependencies (change on lockfile update), then application code (changes every commit). This maximizes cache hits — 90% of builds reuse cached dependency layers.
4. Run as non-root with read-only root filesystem. USER 1000:1000, securityContext.runAsNonRoot: true, readOnlyRootFilesystem: true, drop all Linux capabilities (drop: [ALL]). One container escape vulnerability = root on the host node.
5. Set resource requests and limits on every container. requests = guaranteed minimum, limits = hard ceiling. CPU limits at 2-3× requests for burstable workloads. Without limits, one memory-leaking pod OOM-kills every pod on the node.
6. Pin images by SHA256 digest, never :latest. :latest is a mutable pointer — two replicas started 10 seconds apart can run different versions. CI should auto-replace tags with digests. Production deployments reference immutable digests only.
7. Liveness ≠ Readiness. Liveness = is the process alive? (fast /healthz). Readiness = can the process serve traffic? (checks dependencies). A liveness probe checking external dependencies kills healthy pods during transient network blips.
8. NetworkPolicy deny-all with explicit allows. Default-deny ingress/egress. Only allow specific namespaces/labels. Without network policies, a compromised frontend pod can reach the database directly — your service mesh isn't a firewall.
9. Use operators for stateful workloads, never raw StatefulSets alone. PostgreSQL, Kafka, Redis in K8s need operators (Zalando, Strimzi, Redis Operator) for backup, failover, upgrades. A StatefulSet without an operator is just a pod with a sticky identity — no managed lifecycle.
10. Image scanning in CI, not as a dashboard. Trivy/Grype blocks builds on CRITICAL/HIGH CVEs with a fix available. Scan deployed images weekly and auto-create Jira tickets for newly discovered CVEs. A scan dashboard nobody reads is security theater.

Error Recovery (STANDARD)

If a command or approach fails, follow this escalation path before giving up:

Hard failure boundary: If 3 different approaches all fail, STOP. Do not iterate infinitely. Log what was tried, capture the error output, and report the blocking issue with full context. Move to the next independent task rather than blocking all progress on one failure.

Cross-Skill Coordination

What good looks like: Docker image builds in under 5 minutes and is under 200MB. Kubernetes manifests pass kubeval validation. Pod startup time < 10 seconds. Liveness and readiness probes configured on every deployment. Resource requests and limits set on every container.

Proactive Triggers

• Docker image build time exceeds 10 minutes → Layer cache is likely broken. Check: are COPY . . instructions placed before RUN npm install? Reorder layers so dependencies install before application code copy. Cache miss on dependency layer = full rebuild. 🔴
• Container running as root in production → USER directive missing from Dockerfile. This is a security incident waiting to happen — root container escape = root on host. Add USER 1000:1000 and securityContext.runAsNonRoot: true. 🔴
• Pod restarting every 30 seconds — liveness probe failing → Check if liveness probe uses the same endpoint as readiness probe. During traffic spikes, the endpoint slows down and K8s kills healthy pods. Liveness = /healthz (fast). Readiness = /ready (service health). 🟠
• Image tag :latest found in production manifest → latest is a mutable tag — what you deployed yesterday is not what you're running today. Pin images by SHA256 digest. CI should auto-replace tags with digests in deployment manifests. 🔴
• No resource limits on production Deployment → A memory leak in one pod can OOM the entire node, cascading to other workloads. Set resources.limits.memory and resources.requests.cpu for every container. Without limits, one bad deploy takes down the cluster. 🔴
• Helm release stuck in pending-upgrade for > 5 minutes → Helm hooks are likely hung. Check helm history <release> and kubectl get jobs -l helm.sh/hook. Hung pre-upgrade hook = blocked deployment. Add helm.sh/hook-delete-policy: before-hook-creation to clean up failed hooks. 🟡
• NodePort/port 80 exposed to public internet without TLS → Ingress/load balancer exposing plain HTTP. Use cert-manager to auto-provision Let's Encrypt certificates. Add ingress.kubernetes.io/force-ssl-redirect: "true" annotation. 🟠
• docker-compose secrets in git repo → .env file committed with database passwords, API keys. Add .env to .gitignore. Use docker-compose secrets or environment variable injection from CI/CD. Rotate exposed credentials immediately. 🔴

State Log

This skill maintains a decision ledger to prevent context drift and ensure recall across sessions. Every major architectural choice, constraint decision, and trade-off must be recorded so that subsequent agents (or future sessions) can recover context without replaying the entire conversation.

Production Checklist (STANDARD)

1. Dockerfiles use multi-stage builds — build stage: full SDK, compile/test. Runtime stage: minimal base (distroless/slim/scratch), COPY --from=build, no compilers or dev headers.
2. Containers run as non-root — USER 1000:1000, securityContext.runAsNonRoot: true, readOnlyRootFilesystem: true, allowPrivilegeEscalation: false, capabilities dropped (drop: [ALL]).
3. Resource requests AND limits set on every container — resources.requests for scheduler, resources.limits for hard ceiling. CPU limits at 2-3× requests. Memory limits with OOM-kill buffer. ephemeral-storage limits prevent log-disk exhaustion.
4. Images pinned by SHA256 digest — no :latest in production manifests. CI auto-replaces tags with digests. Image pull policy IfNotPresent with digest tags prevents registry outage from blocking restarts.
5. Liveness and readiness probes configured — liveness = fast /healthz (process alive), readiness = /ready (dependencies checked). Differentiated endpoints, different timing. Liveness never checks external dependencies.
6. NetworkPolicy default-deny with explicit allows — deny-all ingress/egress baseline. Only allow required namespace/label pairs. No open 0.0.0.0/0 rules without justification.
7. PodDisruptionBudget set — minAvailable or maxUnavailable ensures voluntary disruptions (node drain, cluster upgrade) don't cause downtime. PDB must not block cluster autoscaler scale-down.
8. Image scanning blocks CRITICAL/HIGH CVEs — Trivy/Grype in CI fails build on fixable CRITICAL. Registry push policy rejects unscanned images. Deployed images re-scanned weekly with auto-Jira for new CVEs.
9. Helm hooks have delete policies — helm.sh/hook-delete-policy: before-hook-creation prevents hung hooks from blocking upgrades. Hook timeouts set. Hook jobs not left orphaned.
10. Ingress terminates TLS — cert-manager auto-provisions Let's Encrypt certificates. force-ssl-redirect: true annotation. Minimum TLS 1.2. HSTS headers configured.
11. etcd backups automated and tested — Velero or cloud-provider backup for cluster state + PV snapshots. Backup restore tested monthly in staging cluster. Backup retention policy documented.
12. Cluster autoscaling configured and tested — HPA for pods, cluster autoscaler for nodes. Scale-down tested (non-disruptive). Overprovisioning buffer for burst capacity. Node cordon/drain verified clean.
13. Monitoring covers Kubernetes + containers — Prometheus node-exporter, kube-state-metrics, cadvisor/container metrics. container_cpu_cfs_throttled_seconds_total monitored — any non-zero indicates CPU throttling.
14. Secrets managed externally, never in K8s Secret objects — External Secrets Operator, Sealed Secrets, or Vault agent injector. Kubernetes Secrets are base64-encoded, not encrypted. Etcd encryption at rest enabled as defense-in-depth.

What Good Looks Like

Containers are minimal, pinned by SHA256 digest, and run as non-root with all Linux security capabilities dropped.

See references/what-good-looks-like.md for the full quality standard.

Deliberate Practice

Kubernetes mastery is built through controlled destruction. The best K8s engineers have broken clusters in every possible way — in sandboxes, not in production.

graph LR
    A[Deploy a workload to a cluster] --> B[Break something: drain a node, kill a pod, exhaust resources]
    B --> C[Observe: did self-healing work? what surprised you?]
    C --> D[Document the failure mode. Add to your mental model of K8s.]
    D --> A

The One Highest-Leverage Activity: Once a month, break your staging cluster in a way you've never broken it before. The failure mode you discover is the one that would have caused a P1 incident in production. Fix the gap before it finds you.

Anti-Patterns

• Docker COPY . . includes .git, node_modules, .env, and everything in .dockerignore that you forgot to exclude. Image size balloons and secrets leak into the image layer history (visible via docker history).
• docker build --no-cache rebuilds every layer, but doesn't pull updated base images. If your FROM node:18 was cached 3 months ago, --no-cache rebuilds on the 3-month-old base. Use --pull to get the latest base image.
• Kubernetes resources.requests without limits means the pod can burst to the node's entire capacity. One memory-leaking pod can OOM-kill every other pod on the node because there's no upper bound. Always set both.
• kubectl apply -f merges the manifest with the live object using strategic merge patch. Deleting a field from the YAML does NOT delete it from the live object — you need kubectl replace or a specific patch to remove fields.
• Readiness vs Liveness probes: if the readiness probe fails, the pod is removed from Service endpoints but stays running. If the liveness probe fails, the pod is KILLED and restarted. A liveness probe that's too aggressive (checking external dependencies) restarts healthy pods during transient network blips.
• Image tag :latest in Kubernetes with imagePullPolicy: Always pulls whatever latest currently resolves to. Two replicas started 10 seconds apart can run different image versions if a new build pushed during that window.
• CrashLoopBackOff pods filling node disk with logs — a misconfigured deployment crashes every 2 seconds, writing a 50KB stack trace to stdout each time. Kubernetes retains the pod's logs for the pod's lifetime. After 8 hours, a single CrashLoopBackOff pod generates 720MB of logs. With 10 such pods on a node with 20GB of ephemeral storage, the disk fills, kubelet can't write to its own logs, and the node becomes NotReady — evicting all healthy pods on that node. Total cost: $15K-$75K per incident in cascading node failures, evicted production workloads, and engineering time spent diagnosing "mystery node pressure." Fix: Set spec.containers[].resources.limits.ephemeral-storage on every container; configure container log rotation with containerLogMaxSize and containerLogMaxFiles in kubelet config; deploy a CrashLoopBackOff alert (Prometheus: rate(kube_pod_container_status_restarts_total[5m]) > 0.05) that pages on-call after 30 restarts.
• CPU throttling from limits set too close to requests — you set resources.requests.cpu: 500m and resources.limits.cpu: 500m (or worse, no limits at all with a low request). The kernel's CFS quota throttles the container whenever it bursts above 500m. Your P99 latency goes from 8ms to 400ms because every request triggers a throttle period. The application appears "slow" but CPU metrics show 50% utilization — the throttling is invisible at the pod level. Your 20-person engineering team spends 3 weeks debugging "the mysterious latency" across microservices. Total cost: $30K-$100K in wasted engineering time and degraded customer experience from invisible CPU throttling. Fix: Set CPU limits at least 2-3x CPU requests for burstable workloads; monitor container_cpu_cfs_throttled_seconds_total in Prometheus — any non-zero value indicates throttling; use the Vertical Pod Autoscaler in recommendation mode to find appropriate values before enforcing them.
• Kubernetes LoadBalancer service left running in a dev cluster over the weekend — a developer creates a type: LoadBalancer service on Friday to test an API endpoint externally. The cloud provider provisions an application load balancer at $0.0225/hour ($16.20/month). They forget to delete it. Six m

*Truncated - read the full file at https://github.com/zeroes-ones/Skills/blob/a20a35e2eef7136945ce72e79af2b83a3193b59f/skills/07-devops/docker-kubernetes/S