Imported from jeremylongshore/tons-of-skills-marketplace (
plugins/saas-packs/castai-pack/skills/castai-prod-checklist/SKILL.md). Install upstream withnpx skills add jeremylongshore/tons-of-skills-marketplace --skill castai-prod-checklist. Copyright stays with the author (MIT).
CAST AI Production Checklist
Overview
Complete checklist for enabling CAST AI cost optimization on a production Kubernetes cluster. Covers Phase 1 (monitoring) through Phase 2 (full automation) with validation steps at each stage.
Prerequisites
- CAST AI tested on a staging cluster first
- Production API key (Full Access)
- Change management approval for node lifecycle changes
Instructions
Complete the phases in order and preserve evidence for every checked item. Start in monitoring-only mode, verify the production cluster identity and baseline metrics, then request the approved change window before enabling automation. Use a two-person review for capacity limits, disruption budgets, and the emergency-disable procedure; do not copy staging keys, policies, or test evidence into the production record without revalidation.
Phase 1: Monitoring Only
- Agent installed with read-only key
- Agent pod healthy:
kubectl get pods -n castai-agent - Console shows cluster as "Connected"
- Savings report generating (wait 24h for full data)
- Review savings estimate before enabling automation
Phase 2: Autoscaling Enabled
- Full Access API key provisioned and stored in secrets manager
- Cluster controller installed
- Evictor installed with conservative settings (non-aggressive)
- Spot handler installed for graceful interruption handling
- Autoscaler policies configured with appropriate limits:
-
clusterLimits.cpu.maxCoresset to safe ceiling -
unschedulablePods.headroomconfigured (10-15%) -
nodeDownscaler.emptyNodes.delaySeconds>= 300 for production -
spotInstances.spotDiversityEnabled= true
-
- Node templates created for workload-specific needs (GPU, high-memory)
- PodDisruptionBudgets set on all critical workloads
Workload Autoscaler
- Workload autoscaler installed
- Critical deployments annotated with min/max resource bounds
- Anti-shrink cooldown set (300s minimum)
- Memory headroom >= 20% for production workloads
Security
- API key in secrets manager (not Helm values files)
- Kvisor security agent installed
- Network policies applied to castai-agent namespace
- RBAC reviewed and minimized
- Key rotation scheduled (90-day interval)
Monitoring and Alerting
- Alert on agent pod restarts:
kube_pod_container_status_restarts_total{namespace="castai-agent"} - Alert on API errors in agent logs
- CAST AI console email notifications enabled
- Savings report reviewed weekly
- Dashboard tracking spot vs on-demand node ratio
Rollback Procedure
# Disable autoscaling immediately (keeps agent monitoring)
curl -X PUT -H "X-API-Key: ${CASTAI_API_KEY}" \
-H "Content-Type: application/json" \
"https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/policies" \
-d '{"enabled": false}'
# Or remove all CAST AI components
helm uninstall castai-evictor -n castai-agent
helm uninstall cluster-controller -n castai-agent
# Keep the agent for monitoring if desired
Error Handling
| Condition | Response |
|---|---|
| Agent offline or API authentication fails | Keep automation disabled; verify secret reference, RBAC, and egress before retrying. |
| Policy response differs from the approved limits | Stop rollout, restore the prior policy, and reopen change review. |
| Eviction or latency alert fires | Disable autoscaling using the tested path and engage the workload owner. |
| Rollback command cannot be exercised safely | Do not proceed to go-live; repair the runbook and test it in staging. |
Validation Commands
# Final pre-go-live verification
echo "=== CAST AI Production Validation ==="
# Agent healthy
kubectl get pods -n castai-agent -o wide
# All components running
helm list -n castai-agent
# Policies correct
curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
"https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/policies" \
| jq '{enabled, unschedulablePods: .unschedulablePods.enabled, downscaler: .nodeDownscaler.enabled, spot: .spotInstances.enabled}'
# Savings estimate
curl -s -H "X-API-Key: ${CASTAI_API_KEY}" \
"https://api.cast.ai/v1/kubernetes/clusters/${CASTAI_CLUSTER_ID}/savings" \
| jq '{monthly: .monthlySavings, percent: .savingsPercentage}'
Output
Create a production-readiness record that ties each checklist item to its evidence, accountable owner, approval, validation timestamp, and tested rollback. A healthy agent alone does not authorize go-live: policy limits, disruption controls, monitoring, and the emergency-disable path must all be confirmed against the intended production cluster.
Examples
For a production launch, capture a redacted policy response, current Helm release state, agent health, alert test, and savings-baseline review in the change record. If an approval or rollback test is missing, leave autoscaling disabled and resolve that gap before enabling it for production workloads.
Resources
Next Steps
For version upgrades, see castai-upgrade-migration.