Instruction file imported from brightlikethelight/steg-cot-organisms (
.cursor/rules/runpod_debug.mdc). Copyright stays with the author.
RunPod Debug Guide
Connection
Check docs/SESSION_STATE.md for current pod IPs and ports — they change on restart.
# Pattern (fill in from SESSION_STATE.md):
ssh root@<IP> -p <PORT> -i ~/.ssh/runpod_key
# NEVER use proxy SSH (ssh.runpod.io) — PTY issues
# SCP scripts to pod instead of SSH heredocs (quoting nightmares)Quick Status
# GPU status
nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv
# Training process
ps aux | grep train_grpo | grep -v grep
# Latest log
tail -20 /workspace/steg-cot-organisms/nohup_*.out
# Errors
grep -E "(Error|Exception|OOM|NaN)" /workspace/steg-cot-organisms/nohup_*.out | tail -5Common Issues
| Issue | Symptom | Fix |
|---|---|---|
| OOM | CUDA out of memory |
Reduce num_generations or batch_size |
| Accuracy collapse | <60% sustained | Penalty too strong, reduce w_penalty |
| Entropy collapse | 16/16 identical completions | Must restart from scratch |
| NaN loss | loss: nan |
Lower LR, increase grad clipping |
| Disk full | No space left |
df -h /workspace, delete old outputs |
| W&B stale | Metrics not updating | wandb sync or check API key |
Key Metrics
| Metric | Healthy | Warning |
|---|---|---|
| accuracy | >85% | <60% = collapse |
| entropy | >0.1 | <0.03 = collapse |
| reward_std | >0 | =0 = no GRPO signal |
| banned_rate | Decreasing | Flat = not learning |
| cot_length | 50-400 tokens | <20 = avoidance |
Emergency
pkill -f train_grpo # Kill training
python3 -c "import torch; torch.cuda.empty_cache()" # Clear GPU
df -h /workspace # Check disk