Instruction file imported from rompoggi/smth2smth (.cursor/rules/start-resume-runs.mdc). Copyright stays with the author.

Starting and resuming runs (mandatory)

Use with distributed run log (.cursor/rules/distributed-run-log.mdc). After a healthy start, update report/$(whoami)/$(hostname).md and tell the user clearly where to look.

Launch stack

• Always nohup for long jobs (unless user asks foreground debug).
• Always uv run + PYTHONPATH=src + PYTHONUNBUFFERED=1 (or python -u).
• Prefer existing scripts/launch_*.sh when they match the experiment — but do not trust default LOG paths (many scripts still write to flat logs/ at repo root).
• Before launch: read the script (or its echoed log / pid lines) and confirm the resolved log path matches logs/{track_a|track_b|study}/ below. If not, override LOG (and PIDFILE if any) on the command line, or wrap with explicit paths — do not change launch scripts unless the user asks.
• After nohup, verify the process is actually writing to that path (tail the intended file, not an old flat logs/*.log).

Run and log naming

Log directory (logs/)

Resolve {dir} from Hydra track= / experiment:

mkdir -p the directory before nohup.

Log file header (write first)

Before or as the process starts, ensure the log begins with a short ASCII header (survives tail -f):

# run: mae150-ft-f4
# started: 2026-05-25T01:50:00+02:00
# track: a
# experiment_doc: experiments/new_ideas_tracka.md
# hydra: experiment=track_a_videomae_official_ssv2_ft

• started: ISO-8601 from date -Is.
• experiment_doc: repo-relative path to the experiments/*.md spec (read-only; no line anchors). Match tags/topic in that doc when useful.
• Do not put non-ASCII in the header.

Example prepend when launching:

LOG=logs/track_a/mae150-ft-f4_20260525.log
mkdir -p "$(dirname "$LOG")"
{
  echo "# run: mae150-ft-f4"
  echo "# started: $(date -Is)"
  echo "# track: a"
  echo "# experiment_doc: experiments/new_ideas_tracka.md"
  echo "# hydra: experiment=track_a_videomae_official_ssv2_ft"
} >>"$LOG"
# then nohup ... >>"$LOG" 2>&1 &

PID files (minimal use)

In the markdown run log, link the pid file only when it exists and the run is still RUNNING; omit after cleanup.

W&B

Resume after crash / VM reboot: keep the same training config (checkpoint, optimizer, scheduler, LR policy as the interrupted run). Append to the same log file.

• Prefer WANDB_RESUME=allow + WANDB_RUN_ID=<id> (id from the previous log’s [wandb] line or W&B UI). Same entity/project as the original run.
• Do not start a fresh W&B run for a continuation unless the user asks.
• Training checkpoint: training.resume_from=<last.pt> (and matching Hydra flags the prior run used). Do not change LR/scheduler unless the user requests a new phase.

(Code may not yet pass resume into wandb.init; env vars are the default until extended.)

Logging health check (required after start)

Wait ~30–90s (or ≥2× log_interval_steps batches), then inspect the log. Fail the launch if any of:

• No step lines while training should run (check training.log_interval_steps in preset).
• One step per line broken: blank line between every step, or multiple step on one line.
• Mojibake / non-ASCII in training lines (config dump may be large; focus on [HH:MM:SS] step / Epoch lines).
• Traceback or CUDA OOM at start.

Healthy pattern (example):

[01:52:07] step 1150/5625 | avg train loss ...

Checks:

tail -40 "$LOG"
LC_ALL=C grep -n 'step .*step' "$LOG" | tail -5    # multiple steps on one line?
LC_ALL=C grep -P '[^\x00-\x7F]' "$LOG" | grep -E 'step |Epoch' | head -3

If unhealthy: stop the process, tell the user what failed, ask whether to truncate/reuse the same log after fixing (flush/PYTHONUNBUFFERED, config, etc.).

Failed boot vs real restart

After a healthy start (agent → user message)

Include explicitly:

1. Log path (markdown link in chat): [`logs/track_a/mae150-ft-f4_20260525.log`](logs/track_a/...)
2. Healthy: yes/no (one line: e.g. “step logging OK at interval 25”).
3. W&B: markdown link to run URL if training and line present; else “n/a (submit/short job)”.
4. Monitor: tail -f … and, if pid file exists, `kill -0 $(cat …pid)`

Then prepend/update the distributed run log (.cursor/rules/distributed-run-log.mdc) with:

• Log: markdown link to the log file (../../logs/track_a/... from report/{user}/{host}.md).
• W&B: markdown link when applicable.
• PID: only if a pid file is in use and run is RUNNING.

Do not

• Edit experiments/** or shared report/track_*.tex when only launching/logging (see distributed-run-log).
• Leave stale .pid files after a run ends.
• Invent logs/e1_host.log-style names.
• Assume scripts/launch_*.sh already uses logs/track_a/ — verify every time; scripts are not authoritative until updated separately.