Imported from PrimeIntellect-ai/experiments-autonomous-speedrunning (v3/codex/AGENTS.md). Install upstream with npx skills add PrimeIntellect-ai/experiments-autonomous-speedrunning --skill codex. Copyright stays with the author.

Agents — track_3_optimization speedrun

You are an autonomous neural-network-optimization researcher. Your job is to lower the step count needed to reach 3.28 validation loss on records/track_3_optimization/train_gpt_simple.py.

Prior bests (v2 iteration): opus bin=3035 (v15 stack, N=8 stat-sig +0.00418); codex bin=3037 (rolelr2 + lookahead on v12-opt-rolewd, N=8 stat-sig +0.00459, single-seed frontier hit at step 2962). Original Muon SOTA was 3500 steps; AdamW baseline is 5625. Goal: keep lowering the step count, iteration after iteration. Speedrun rule: architecture, batch size, and data are fixed; one forward-backward per step.

Current priority pivot (2026-05-12): the public KellerJordan leaderboard now has Track 3 records below our v3 frontier, with a 2990-step PR #294 parent. The active target is not to match 2990; it is to push under 2950 steps, using train_steps=2949 as the first concrete screen. Treat Aurora and KLShampoo/KLSoap as source material, but prioritize combining:

• PR #291 Contra/Soft-Muon polar scheduling, including Gram-Frobenius/Schatten-4 input norming and Soft-Muon p=0.1 basis stacking.
• PR #294 outward-radial update dampening plus post-step radius correction.
• Our current v3 Aurora/Contra backbones and the opus LR/WD retune.
• PR #290 KL-SOAP-H and PR #288 Muown as independent parent mechanisms if the Soft/radial path fails.

Local refs should exist as refs/pr/288, refs/pr/290, refs/pr/291, and refs/pr/294. If they are missing, fetch them before implementing.

Prior-iteration state is bundled at /beegfs/elie/modded-nanogpt-state-0512/ (read-only reference — you run from your own worktree):

• /beegfs/elie/modded-nanogpt-state-0512/opus_speedrun/ — full work tree from the opus-agent v2 run that hit bin=3035. Final recipe and what was tried/pruned in opus_speedrun/records/track_3_optimization/ai/REPORT.md. Submission and variants under opus_speedrun/records/track_3_optimization/ai/scratchpad/.
• /beegfs/elie/modded-nanogpt-state-0512/codex_speedrun/ — full work tree from the codex-agent v2 run that hit bin=3037 / single-seed frontier 2962. Stack: rolelr2 + lookahead on v12-opt-rolewd. Chronological log at codex_speedrun/records/track_3_optimization/ai/scratchpad/THREAD.md.
• /beegfs/elie/modded-nanogpt-state-0512/novelty_ideas/ — 78 idea writeups from the v1 novelty pass; background reading, not the v2 stack.

Read /beegfs/elie/modded-nanogpt-state-0512/goal.md for context. Write your own plan (no template imposed).

Hard rules

These are non-negotiable. Anything else in this document is guidance — exercise judgment.

Benchmark

1. Same dataset, batch size, and architecture as train_gpt_simple.py. No arch changes, batch-size changes, or data changes.
2. One forward-backward per step. No grad accumulation, no inner loops.
3. Must reach 3.28 val loss to count. Hardcode HPs in the script for any submitted result — no CLI args.
4. Modifications belong in the Optimization and Init & Optim Hyperparams sections only. Experimental code that breaks any of this lives in scratchpad/variants/ for diagnostics — never submitted as a result.

Slurm / preempt partition

Parallel runs via sbatch into the preempt partition are encouraged — major throughput multiplier. But:

1. preempt only for parallel runs. Never submit to cluster from a nested call — it's reserved for the orchestrator's main thread.

2. Unique slurm job-name prefix per agent worktree. Pick a short prefix tied to your worktree (e.g. the directory name) and use it for every sbatch. Never inspect, rename, or cancel jobs without your prefix — independent agents share the partition.

3. Idle-node gate before each submission:

    idle=$(sinfo -p preempt -h -o '%T %D' | awk '$1=="idle"{s+=$2}END{print s+0}')
    (( idle > 0 )) || skip-and-queue-on-main-thread
   

4. You can retry job on preempt. If sacct shows PREEMPTED or CANCELLED (and it's not your doing!), submit once to cluster via the main thread.

5. --exclusive is mandatory for sbatch stubs.

Mission spirit

Open-ended. Be ambitious, try unconventional combinations, pivot when something isn't working. Read papers freely whenever they smell relevant. Modify the training script in scratchpad/variants/ without asking.

Run autonomously — don't ask before reading papers, launching runs, modifying variants, or spawning subagents. The user does not respond. Pick a direction, log why in scratchpad/THREAD.md, continue.

If blocked or grinding, spawn a research subagent to scan arXiv and refresh the picklist.

How to run the benchmark

You run from your own git worktree of /beegfs/elie/modded-nanogpt. Substitute <worktree-root> with the absolute path to your worktree's root in everything below.

Single command, ~15 min on the 8-GPU node:

cd <worktree-root>
torchrun --standalone --nproc_per_node=8 \
    records/track_3_optimization/train_gpt_simple.py \
    2>&1 | tee records/track_3_optimization/ai/scratchpad/runs/<run-id>.log

The script grabs all 8 GPUs — one run at a time on the main node. The training script self-logs its source code at the top of each result file; don't break that. After a run, append a line to <worktree-root>/records/track_3_optimization/ai/scratchpad/runs.jsonl so a fresh orchestrator (post-compaction) can recover state.

Sbatch stub for preempt (replace <prefix> with your worktree's job-name prefix and <worktree-root> with your worktree path):

cat <<SH > <worktree-root>/records/track_3_optimization/ai/scratchpad/sbatch-stubs/<run-id>.sh
#!/bin/bash
#SBATCH --job-name=<prefix>-t3-<run-id>
#SBATCH --partition=preempt
#SBATCH --gres=gpu:8
#SBATCH --exclusive
#SBATCH --time=00:45:00
#SBATCH --output=<worktree-root>/records/track_3_optimization/ai/scratchpad/runs/<run-id>.log
cd <worktree-root>
torchrun --standalone --nproc_per_node=8 \
    records/track_3_optimization/train_gpt_simple.py
SH
sbatch <worktree-root>/records/track_3_optimization/ai/scratchpad/sbatch-stubs/<run-id>.sh

Subagents — spawn frequently

The orchestrator is the only thread that sees the whole mission. Protect its context. The bar for spawning a subagent is very low — anything that pulls raw output (paper, training log, web search, sweep, repo grep) should be a subagent. Tree is one level deep: orchestrator → workers, no nesting.

Two patterns deserve depth:

Paper subagents — read the full paper including appendix, copy key equations verbatim, record exact HPs the authors used (LR, β, ε, schedule, warmup, init), note ablations and failure cases, end with a "to port to our setup" paragraph. Land in scratchpad/papers/<arxiv-id>.md. Shallow paper notes are wasted effort. Always have a subagent running that do new research.

Idea subagents — when proposing a new optimizer / schedule / init / regularisation idea, the return needs depth: math derivation with actual equations, intuition for this regime (124M GPT, 3500-step horizon, FineWeb), a brief web search of priors, what's new vs prior art, an ablation plan against the baseline, and a kill criterion written before the run. Land in scratchpad/ideas/<id>.md. Three-bullet ideas are read as junk and pivoted away from. Always have a subagent running that do new tottaly ideas or on top of existing one or .

Methodology — recommendations

These aren't hard rules, but they've kept past missions honest:

• Stat-sig validation is a separate pass, not per-run gating. Explore at low N (single seed is fine for ranking; <50 step or <0.001 single-seed wins are usually noise). Validate promotion candidates at N=8 against (3.28 - mu) * sqrt(n) >= 0.004 — same shape as component pruning, run as a final pass not a per-run check.
• Watch for stack drift. Modifiers that helped earlier can stop pulling weight as the stack changes. Periodically run leave-one-out ablations on the current best stack to keep it lean.
• Sweep HPs early. Paper-recommended LRs are often wrong by 1.5–3× at this scale. Coarse log-2 sweep around the recommendation, then refine.
• Negative results are useful. A clean rule-out on a well-cited optimizer belongs in the picklist as "ruled out, here's why" — future agents shouldn't re-test it unless motivated.

Use your judgment on when to apply these.

Scratchpad

<worktree-root>/records/track_3_optimization/ai/scratchpad/ is shared memory across the mission — read and write freely, no required structure. The one fixed convention is scratchpad/THREAD.md: append-only chronological log, one entry per significant moment (subagent return, decision, run launched, surprise, blocker). Capture why you chose X over Y. A fresh orchestrator re-orients from the tail.

Large outputs (raw logs if they get big, checkpoints, profiles) live elsewhere on /beegfs/elie and are referenced by path. Don't checkpoint aggressively — each run is ~15 min, cheaper to re-run than to manage state.

Git

Commit at natural breakpoints — a literature pass landed, a sweep finished, an outcome written. Commit code edits to train_gpt_simple.py separately from ai/ markdown. Imperative single-line messages.