Imported from junhyekh/rlxp (plugins/rl-experiment-assistant/skills/rlxp-autoloop/SKILL.md). Install upstream with npx skills add junhyekh/rlxp --skill rlxp-autoloop. Copyright stays with the author.

RLXP Autoloop

Use this skill when the user asks RL Experiment Assistant to continue, resume, or fully automate an RL experiment loop from existing .rlxp/ state. The agent owns the operational work: read state, validate gates, launch approved runs only through scripts/rlxp_autoloop_runner.py, parse metrics, update artifacts, and keep iterating until the approved budget or stop rule is reached.

Do not depend on chat history. At the start of every invocation, infer the current state from the target repository.

Required state

Read .rlxp/registry.yaml first. If it contains multiple tasks and no active task/study is selected, stop and ask which task to operate on. Never mix artifacts across tasks or studies unless the user explicitly asks for a cross-study comparison.

Read these files before deciding anything:

• .rlxp/registry.yaml
• .rlxp/tasks/<task_id>/task_card.md
• .rlxp/tasks/<task_id>/adapter.yaml
• .rlxp/tasks/<task_id>/studies/<study_id>/contract.yaml
• .rlxp/tasks/<task_id>/studies/<study_id>/state.json, if present
• .rlxp/tasks/<task_id>/studies/<study_id>/experiments.yaml, if present
• .rlxp/tasks/<task_id>/studies/<study_id>/ledger.jsonl, if present
• .rlxp/tasks/<task_id>/studies/<study_id>/report.md, if present
• .rlxp/tasks/<task_id>/studies/<study_id>/candidates/, if present
• .rlxp/tasks/<task_id>/studies/<study_id>/runs/, if present

Set <study_root> to .rlxp/tasks/<task_id>/studies/<study_id>/. For legacy repositories without .rlxp/registry.yaml, use the root .rlxp/ files as a single implicit task/study and set <study_root> to .rlxp/.

Use helper scripts from this plugin when they fit:

• scripts/rlxp_read_state.py
• scripts/rlxp_check_budget.py
• scripts/rlxp_validate_candidate.py
• scripts/rlxp_autoloop_runner.py, only for the launch transition
• scripts/rlxp_append_ledger.py
• scripts/rlxp_render_report.py
• scripts/rlxp_score.py

These helpers are implementation details for the agent. Do not ask the user to run them by hand unless the agent cannot execute local commands. All helpers except rlxp_autoloop_runner.py are dry-run or artifact-only helpers and must not start training, eval, GPU, W&B, simulator, or network work.

Hard gates before training

Before any GPU-consuming command:

1. Re-read <study_root>/contract.yaml.
2. Require contract status: approved_for_launch.
3. Require automation.mode: autonomous, or an equivalent explicit user approval recorded in the contract.
4. Require remaining iteration, wall-clock, and GPU-hour budget.
5. Require the candidate lifecycle to be validated or approved_for_launch after candidate validation.
6. Require the candidate to be inside the approved tuning scope.
7. Require the primary metric and evaluation protocol to be unchanged.
8. Record launch command, seed, GPU allocation, run ID, and expected outputs before execution.
9. Launch only through scripts/rlxp_autoloop_runner.py with --execute --confirm RUN_CANDIDATE.
10. Stop if the next useful action requires an out-of-scope change.

An approved experiment entry is not sufficient by itself. The contract, budget, scope, metric, evaluation protocol, runner execution flag, and confirmation token must still pass immediately before launch.

Autonomous loop

Repeat until a stopping rule fires:

1. Infer current state from <study_root>/ledger.jsonl, <study_root>/state.json, and <study_root>/runs/.
2. If no valid contract exists, draft one and stop before training.
3. If automation is not approved, stop before training and ask for approval.
4. If no valid baseline exists, prepare and run only the approved baseline.
5. Collect metrics from approved metric sources.
6. Compare the latest run to the incumbent using the contract primary metric and guardrails.
7. Classify the result as accept, reject, inconclusive, or debug.
8. Diagnose the dominant failure mode.
9. Propose the next candidate inside the approved scope.
10. Validate the candidate before launch.
11. Launch only validated or approved candidates through scripts/rlxp_autoloop_runner.py.
12. Update <study_root>/ledger.jsonl, <study_root>/state.json, <study_root>/report.md, and run-local summaries.
13. Continue automatically unless a stop rule or approval boundary is reached.

Candidate priority

Use this default intervention order:

1. instrumentation, metric extraction, or baseline repair
2. reward parameter tuning
3. curriculum or adaptive sampling adjustment
4. domain-randomization adjustment
5. reward engineering only if justified by evidence and allowed by contract

Represent each candidate as a hypothesis with expected effects, risks, and an explicit decision rule. Never accept a candidate solely because training reward increased.

Candidate lifecycle

Track each candidate through draft, validated, launched, completed, accepted, rejected, inconclusive, debug, and when applicable reverted. A candidate may launch only after validation has made it validated or approved_for_launch, and the only bundled transition to launched is rlxp_autoloop_runner.py. Acceptance requires the result audit to support the metric claim, not just a favorable report row.

Prefer simple candidates when scores are similar. Record simplicity costs such as patch lines, changed files, new reward terms, and new hyperparameters so the incumbent does not drift toward unnecessary complexity.

Practical evidence routines

• Reward-related runs should produce or consume <study_root>/runs/<run_id>/reward_reflection.json, summarizing task metrics, reward-component trends, saturation, domination, and hacking risks before the next reward proposal. Reward engineering requires this reflection unless the contract records an explicit exception and the candidate explains why reflection is impossible or irrelevant.
• DR changes should be preceded by a controlled DR probe summary. If no probe exists, usually plan one-parameter perturbation probes before widening ranges. A DR tuning candidate without a probe summary requires an explicit impossibility reason.
• Curriculum work should choose among sampling reweighting, difficulty schedules, and environment generation. Environment candidates must add learnable intermediate cases, bridge the known difficulty gap, preserve diversity, keep the held-out evaluation distribution fixed, and include a rollback rule.
• Parallel batches should isolate code/config edits with one candidate per worktree or equivalent run sandbox, usually under .rlxp/worktrees/, and record GPU, seed, expected output, and promotion criteria.
• Multi-run execution should behave like a queue: smoke, short trials, promoted multi-seed validation, incumbent validation, monitor, audit, and report update.

Decision rule

Accept a candidate only when all are true:

• the approved primary metric improves relative to the incumbent
• guardrail metrics pass
• the evaluation protocol is unchanged
• metric sources are approved
• the candidate stayed within the approved tuning scope

Default classifications:

• accept: primary metric improves and guardrails pass
• reject: no improvement, guardrail regression, instability, or reward hacking
• inconclusive: missing, noisy, or insufficiently separated evidence
• debug: training, logging, evaluation, or metric extraction failed

Stop rules

Stop when any condition holds:

• maximum iterations reached
• GPU-hour or wall-clock budget exhausted
• target score reached
• patience exceeded
• all valid candidates exhausted
• candidate validation repeatedly fails
• metric extraction repeatedly fails
• training repeatedly crashes
• persistent guardrail failure
• next useful action is outside the approved contract
• user approval is required by the contract

Required output after each iteration

Update these artifacts:

• <study_root>/ledger.jsonl
• <study_root>/state.json
• <study_root>/report.md
• <study_root>/runs/<experiment_id>/command.sh
• <study_root>/runs/<experiment_id>/metrics.summary.json, when metrics exist
• <study_root>/runs/<experiment_id>/analysis.md
• <study_root>/candidates/<candidate_id>.yaml, for proposed next candidates

Report the current decision, metric evidence, guardrails, remaining budget, next action, and stop condition.