About

Imported from jeremylongshore/tons-of-skills-marketplace (skills/.curated/coreweave-incident-runbook/SKILL.md). Install upstream with npx skills add jeremylongshore/tons-of-skills-marketplace --skill coreweave-incident-runbook. Copyright stays with the author (MIT).

CoreWeave Incident Runbook

Name: coreweave-incident-runbook
Author: jeremylongshore

Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Overview

Respond to GPU workload incidents by stabilizing customer impact, preserving redacted evidence, and restoring a known-good state. The incident commander owns communications and escalation; responders use only the access required for triage.

Prerequisites

An incident ID, named commander, affected namespace/service, and on-call route.
Authorized read-only cluster access plus a documented production rollback revision.
A redaction policy for logs, prompts, model artifacts, and credentials.

Instructions

Declare severity and scope, then capture pod, event, node, and service status.
Stabilize impact with the documented scale, failover, or rollback action before root-cause work.
Collect only redacted, bounded diagnostics and escalate hardware or capacity issues through CoreWeave support.
Verify recovery against the service SLO, update stakeholders, and create follow-up work for root cause and prevention.

Triage Steps

# 1. Check pod status
kubectl get pods -l app=inference -o wide

# 2. Check recent events
kubectl get events --sort-by=.lastTimestamp | tail -20

# 3. Check node status
kubectl get nodes -l gpu.nvidia.com/class -o wide

# 4. Check GPU health
kubectl exec -it $(kubectl get pod -l app=inference -o name | head -1) -- nvidia-smi

Common Incidents

Inference Service Down

Check pod status and events
If OOMKilled: reduce batch size or upgrade GPU
If ImagePullBackOff: check registry credentials
If Pending: check GPU quota and availability

GPU Node Failure

Pods will be rescheduled automatically
If no capacity: scale down non-critical workloads
Contact CoreWeave support for extended outages

Model Loading Failure

Check HuggingFace token secret exists
Verify model name spelling
Check PVC has sufficient storage
Review container logs for download errors

Rollback

kubectl rollout undo deployment/inference

Output

A time-stamped incident record with scope, owner, stabilization action, and redacted evidence.
A verified recovery or an explicit escalation with a safe customer-impact mitigation.

Error Handling

Incident complication	Required response
Rollback fails	Stop repeated deploy attempts, escalate to the platform owner, and preserve events.
Diagnostics include a secret	Restrict distribution, rotate the secret, and recollect redacted evidence.
No GPU capacity is available	Prioritize critical services under approved policy; do not remove tenant quotas.
Recovery cannot meet SLO	Declare the continuing impact and use the approved fallback or communication path.

Examples

For a failing production rollout, stabilize first and capture only the relevant evidence:

kubectl -n inference-prod rollout undo deployment/inference
kubectl -n inference-prod rollout status deployment/inference --timeout=10m
kubectl -n inference-prod get events --sort-by=.lastTimestamp | tail -30

Record the revision, outcome, and redacted events in the incident. Do not retry a broken image or expose a troubleshooting endpoint publicly.

Resources

CoreWeave Support
CoreWeave Status

Next Steps

For data handling, see coreweave-data-handling.

Use it

Copy one of these into your project. Installing also returns the manifest and these snippets.

yaml

targets:
  - https://api.opensmartroute.ai/api/v1/registry/jeremylongshore-tons-of-skills-marketplace-coreweave-inc-bf4290/manifest   # or paste the manifest below

Manifest

An Open Capability Manifest: the router reads it to know what this does, what it costs and when to pick it.

jeremylongshore-tons-of-skills-marketplace-coreweave-inc-bf4290.ocm.jsonjson

{
  "ocm": "1",
  "id": "jeremylongshore-tons-of-skills-marketplace-coreweave-inc-bf4290",
  "kind": "skill",
  "name": "coreweave-incident-runbook",
  "description": "Incident response runbook for CoreWeave GPU workload failures. Use when inference services are down, GPUs are unavailable, or responding to production incidents on CoreWeave. Trigger with phrases like \"coreweave incident\", \"coreweave outage\", \"coreweave runbook\", \"coreweave service down\".",
  "publisher": "jeremylongshore",
  "version": "1.0.0",
  "capabilities": {
    "domains": [
      "general"
    ],
    "tags": [
      "skill-md",
      "saas",
      "gpu-cloud",
      "kubernetes",
      "inference",
      "coreweave",
      "skills-sh"
    ],
    "languages": [
      "en"
    ]
  },
  "quality_prior": 0.6,
  "examples": [
    "Incident response runbook for CoreWeave GPU workload failures. Use when inference services are down, GPUs are unavailable, or responding to production incidents on CoreWeave. Trigger with phrases like \"coreweave incident\", \"coreweave outage\", \"coreweave runbook\", \"coreweave service down\"."
  ],
  "primary": false,
  "metadata": {
    "source": {
      "provider": "skills.sh",
      "repository": "https://github.com/jeremylongshore/tons-of-skills-marketplace",
      "path": "skills/.curated/coreweave-incident-runbook/SKILL.md",
      "ref": "HEAD",
      "url": "https://github.com/jeremylongshore/tons-of-skills-marketplace/blob/HEAD/skills/.curated/coreweave-incident-runbook/SKILL.md",
      "key": "jeremylongshore/tons-of-skills-marketplace/skills/.curated/coreweave-incident-runbook/SKILL.md"
    },
    "compatibility": "Designed for Claude Code",
    "allowed_tools": [
      "Read,",
      "Bash(kubectl:*),",
      "Grep"
    ],
    "license": "MIT"
  },
  "instructions": "# CoreWeave Incident Runbook\n\n> **Community-contributed.** Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.\n\n## Overview\n\nRespond to GPU workload incidents by stabilizing customer impact, preserving\nredacted evidence, and restoring a known-good state. The incident commander owns\ncommunications and escalation; responders use only the access required for triage.\n\n## Prerequisites\n\n- An incident ID, named commander, affected namespace/service, and on-call route.\n- Authorized read-only cluster access plus a documented producti",
  "cost": {
    "context_tokens": 868
  }
}

Fetch it by URL: GET /api/v1/registry/jeremylongshore-tons-of-skills-marketplace-coreweave-inc-bf4290/manifest?version=1.0.0

Reviews

Star ratings from people who tried it. One review per account; edit yours any time.

No reviews yet. Install it, try it, and be the first to rate it.