Imported from Metta-AI/coworld (templates/optimizers/252026/jboggs-crewrift-player-optimizer/optimization-loop-specific/skills/experiment-discipline/SKILL.md). Install upstream with npx skills add Metta-AI/coworld --skill experiment-discipline. Copyright stays with the author.

Experiment discipline — recipes (loop tier)

On-demand recipes (42). Trigger→action heuristics; pull the relevant one when its situation arises.

1. A bad league/round score is not a policy signal until you rule out config, infra, and role

loop · negative result · ⚠ session-derived, unverified

Before treating a loss as a quality signal, read the round env config and pull the artifacts. A certification smoke config (tasksPerPlayer:0, roleRevealTicks:0, startWaitTicks:0, gameOverTicks:1) resolves to an instant crew win, so a player assigned imposter loses regardless of skill. Discard infra-failed rounds: 'results contain 9 scores for 8 stored policy versions' is a data mismatch, and two different policies both disqualified within ~4 minutes without playing a turn meant the league referee crashed at startup (an IAM role could not read the tournament signing key from S3) — so pull GET /jobs/{job_id}/artifacts/logs before assuming failure on merit. To distinguish 'ran but lost' from 'instant/aborted game', pull the full per-episode artifact set (results, runtime_info, replay, debug, logs, trace, per-agent policy logs); zero kills/tasks/votes across all slots means an aborted game. When scores collapse to a fixed floor (e.g. -10 in every imposter game), suspect a hard penalty for a missed mandatory action (not voting) and check whether the decisive event fired at all before tuning. sources: claude-code:3837ef5e-3e9c-41bc-b290-df5865581698, claude-code:77e158b0-bd16-41f8-8473-d1b1522d451b, codex:019e18ed-f5bd-7083-a170-7833d83d5236, codex:019e1d2c-7707-7223-bd6c-d2c41104a10a

2. Reproduce a vague infra complaint locally before blaming the league, and re-check live state

loop · ⚠ session-derived, unverified

Reproduce a vague 'league games aren't running well' complaint locally first by running the exact uploaded image in a full local tournament-config game; in one case this reproduced the same shape (imposters kill ~twice, one meeting, crew grind to a draw), proving it was a debuggable strategy issue, not flaky league tooling. If it does NOT reproduce in local single-machine self-play, escalate to a hosted self-play game mirroring tournament settings before theorizing, because latency, scheduling, and kick behavior only appear in the hosted runner. Re-resolve volatile facts live rather than trusting stale notes: coworld leagues showed the Cue-n-Woo league existed despite docs saying 'No live league yet'. sources: claude-code:25488cd9-61d5-4231-bd87-e2744b6db043, claude-code:5f9e8287-da92-4dff-bf39-2511bcaf0700, claude-code:77e158b0-bd16-41f8-8473-d1b1522d451b

3. Count whether the decisive event fired before judging how well it was done

loop · ⚠ session-derived, unverified

Ask 'did it do X (count==0)?' before 'did it do X well?': emit a named trace event for every required/decisive action so the zero-count query is trivial, and treat zero variance across a batch (identical score in all 100 episodes) as proof the interesting code path never executed, not as consistency. Real cases: meeting_vote_selected fired 0 times across all games; meeting-LLM events (meeting_start, meeting_llm_decision, meeting_llm_fallback) returned 0 across 100 episodes. Trust wire/server counters (messages, actions, chats, last_chat_chars, server_confirmed) over your build flags - a 'chat+voting enabled' build reported chats=0, and agents logged many OfferRoleExchangeTask events with server_confirmed:false (activity without success). Before concluding an event is truly absent, rule out two false-absence traps: cross-check the actual emit call site to confirm you grepped the right event name, and separate 'subsystem broken' from 'precondition never reached' (dead/ghost agents stop running meeting/vote sub-policies, so a ghost emits none by construction - the agent was dead in 72/100 games). Not every seed yields the event, so grep traces for the trigger (report_attempted/meeting_started) and analyze the run that actually contains it. sources: claude-code:3837ef5e-3e9c-41bc-b290-df5865581698, claude-code:aae25940-ade3-41bc-b817-517dc86ccebf, claude-code:d48f9347-7884-4ea2-8659-230051d86410, claude-code:e5b86992-9b8a-498e-a47a-1ead507cc54c (+1)

4. Confirm the real LLM/provider path fired before reporting any behavior conclusion

loop · ⚠ session-derived, unverified

Missing credentials or a disabled LLM degrade silently into a scripted fallback that runs clean with zero LLM calls and looks fine at the harness level, so verify liveness via LLM-driven artifacts (meeting chat lines, suspicion statements, guidance.jsonl), not frame counts or absence of stderr; treat any credential-missing log ('OPENAI_KEY not set') as proof the LLM path did NOT exercise, and treat a credential-missing/fake-key vs real-key paired short match as a differential probe to isolate a key-gated hang. Confirm WHICH provider is wired before reporting 'LLM behavior' — a heuristic/smoke provider (~82% hold/fallback, order-biased probing, one canned message) tells you nothing about the real model — and confirm the action channel: machina runs logging talk_enabled=False expose only 5 core actions and fall back to change_vibe_* sideband instead of speech-bubble talk, so labeling a run 'talk-mode' without checking the init log and config action count gives a wrong experiment label. Do not silently disable a policy's LLM for local convenience; only disable when explicitly instructed for a stated reason. Pull behavior examples from trace/replay files labeled by seed, never paraphrase from memory, because behavior varies a lot by seed (one seed produced a single chat line; another was chat-heavy). sources: codex:019e131a-1bb0-7400-b54a-f208202b291d, codex:019e14a5-e574-7770-953b-17019f2f5e74, codex:019e184d-2de6-7b22-b8c0-853edbf53265, codex:019e196e-93c2-7523-8da7-96f782ea578c (+1)

5. When a player 'does nothing', read trace volume and mode/phase, not its strategy

loop · negative result · ⚠ session-derived, unverified

A passive/idle agent is usually a wiring or strategy-gate bug, not bad strategy, and standard early-exit signals lie. Empty traces (manifest start_tick:0/end_tick:0, zero JSONL rows) mean step never ran or bailed before emitting; reproduce with a short match and measure action mix (e.g. '100% noop over 2,320 frames') to localize the fault to wrapper vs server vs policy. Do NOT trust end_tick as a death/exit proxy: it only advances when the player emits a game event, so a passive policy looks like an insta-exit at role-reveal while it is alive and stepping for hundreds more ticks (the websocket closes when the game ends, not when the bot stalls - instrument ws-close frame counts). Trace which mode/phase the policy sits in and whether the unlock condition is even reachable within the episode, because a bot can stay stuck in its lowest-aggression mode all match when a gating perception flag never flips. Beware that adding stderr tracing can make a long-standing bug suddenly visible in a code-identical submission - do not blame the latest deploy for symptoms that only became observable then; diff the actual logic. sources: claude-code:4ec865f5-bcbc-44f1-b109-a27d69f55506, codex:019dfea2-76a2-7ba0-996a-9be68ceee04f

6. Green tests never certify a policy: validate every boundary AND climb a ladder to a live traced match, confirming which code path ran

loop · negative result

Tests verify wiring and message shape, not competence — one generated agent passed 24 tests yet sent wrong bytes and ran a fixed right-down-left-up walk ignoring the observation, and a localization fix passed pytest+Ruff yet a full 10k run regressed to zero score. Validate up a ladder where no tier substitutes for the next: (1) compile/import smoke run from the agent's OWN cwd (running run_agent.py from repo root fails 'can't open file'); (2) the targeted unit test using a synthetic frame reproducing the exact condition; (3) the full per-agent suite plus documented guard tests (FFI/action-table) even for guards the change doesn't obviously touch; (4) a traced live multi-agent match against the real server, because only a live run exercises the perception->belief->task-selection render path and surfaces whether multi-step inference fires in sequence. Test each boundary on its own too — perception (fixture/snapshot, per-frame perf, UI/role-reveal/voting/localization regression fixtures), belief update (per-transition, reset/phase-transition, evidence-provenance, stale-vs-current), modes (legality, lifecycle, scratch reset, completion/stalling), action layer (pathfinding, cursor/button, multi-tick holds, movement-failure recovery), strategy (snapshot schema, determinism, directive validation, TTL/watchdog/fallback), and LLM (fake-provider, malformed-output rejection, shadow eval, live trace review before enabling control). Confirm WHICH code path your green tests cover before trusting them: the cogsguard suite passed despite a broken baseline because it tested gear=1 against Planky missions that pre-seed inventory and never need recipe discovery; conversely a failure reproducing across two independent paths (in-process rollout and the Docker websocket image-contract test) is the policy's bug, not a harness artifact. Gate acceptance on whether the agent perceives the LIVE observation and takes task-relevant actions, confirm coverage by counting which action/task types actually fired in the target phase (an agent logged 3600+ error-free ticks while only one task type fired and it never engaged opponents), and assert on decision-trace CONTENT not count (a nav object has a non-empty path / nonzero edge length / positive progress, not merely that it exists). Keep the division of labor straight: unit tests own internal-mechanism correctness, the live run only proves 'the pipeline still flows'; cover a fallback/no-LLM-only path with a direct unit test. When a generated player fails, distinguish a wrong I/O contract (regeneration fixes it) from stub decision logic (it will NOT). 'Wired in but inert' (correct beliefs in stderr while still emitting noop) is a valid milestone that proves perception before strategy depends on it. sources: auggie:5d1ff8d5-2505-4dca-af45-56d3070363e2, claude-code:33356b24-de19-46ae-ad01-810557f4d9b1, claude-code:3bdbae7f-58b2-4e0e-880b-2a2e03cc0aad, claude-code:7b66497b-650c-4bba-b410-6c854f59163d (+21)

7. Re-run the full regression suite after every tweak; prove a change is scoped before trusting it

loop · ⚠ session-derived, unverified · see related: S005 (other tier)

Re-run BOTH the new-feature tests and the full regression suite after EVERY hardening tweak, not once at the end, so the final reported numbers match the actual tree (re-running Stage 8 + the 194-test regression after each compatibility change kept pass counts honest); green NEW tests alone never prove the old loop still holds. Changing a player's per-tick inner-loop contract is high-risk: moving shared universal updates (tick increment, view assignment, cooldown decay) into a new module changes that contract, so re-run prior-stage regressions afterward, and add new pipeline capability via OPTIONAL constructor args (e.g. an optional hook_registry) so earlier versions/tests keep working. Before assuming a change is local, grep the symbol to confirm the call-site count (production had exactly ONE updateTaskState call, which with the no-shield sentinel confirmed the edit was fully scoped). Ship regression tests that reproduce the runtime failure, not the happy path - for a vote-latch fix assert a permanent 404 votes within 2 ticks and latches, the latch carries into later meetings with no further LLM calls, and repeated transient failures still end in a vote. Validate domain tracing end-to-end too: drive a real runtime.step() with a crafted frame that flips a game condition and assert the domain.* event reaches the trace sink, since unit-testing the observer alone does not prove live composition. sources: claude-code:3837ef5e-3e9c-41bc-b290-df5865581698, claude-code:bc36021d-994c-48ec-a3bc-7ec637f7464f, claude-code:c6965c58-2cb9-46fb-8b29-bad3f3e9e77f, codex:019e0140-e554-78e3-b66f-80634cc63f5d (+6)

8. Prove the lever fires before trusting an A/B; byte-identical or candidate-set-of-one results are a DEBUG SIGNAL, not a null result

loop · negative result

Before (or right after the first run of) an A/B, prove the lever you changed actually fires; byte-identical / score-equal-to-baseline results are a DEBUG SIGNAL the change is inert, not a neutral result. When >=4 of 12 (especially >=8 of 12) seeds match baseline exactly at 2000 steps, capture one single-seed trace and confirm the flag's code path executes — far cheaper than another 12-seed sweep, and cheaper than guessing (one session burned two experiment cycles, E018 plus an adjacency arm, on guessed fixes for a seed-256 regression before a per-tick trace+diff microscope localized the real mechanism in ~20 minutes). Diagnose an inert flag by cause: (a) HORIZON — horizon-dependent flags (late-game K=1 collapse, late stale-cap removal, deficit stall) only trigger on extractor depletion, so confirm at 10k steps before a reject is meaningful; (b) ROLE NOT INSTANTIATED — a flag for scramblers in a 4-miner/4-aligner mix is byte-identical; (c) NO ARGMAX CHANGE — a weight/multiplier flag is inert when the base weight already ranks correctly; (d) CANDIDATE-SET <=1 — a sort-key or penalty modification has zero effect when the set it reorders has <=1 member (two aligner danger-penalty experiments were inert because the neutral-junction candidate set was 0-1 at 10k, so the real bottleneck is candidate-set expansion, not selection). Concrete checks: add a counter for how often a constraint rejects a would-be pick before relaxing it; confirm a tie fires on >~5% of ticks before a tie-break; modify the DOMINANT decision path not a rare secondary fire site; verify game-config constants (cargo limits, role gear modifiers, extraction amounts) in authoritative source before sizing thresholds, since effective values dwarf base values once role modifiers apply. When two threshold variants are byte-identical the events are not separable by that parameter, so trace before generating more variants (a 'skip when carrying heart' gate was byte-identical and useless because the causal early no-heart event was upstream of the visible throughput-stall symptom). Any edit to decide.py runs a fixed triad first — compile, the substrate regression suite, the Full spot check on the required three seeds — even for an off-by-default flag, because the dispatch path is shared. sources: alpha_cog/AGENTS.md, claude-code:425dc11e-1c8d-4d12-bf44-adf01b6fa796, claude-code:e049dbf8-9702-41e5-8ac5-830cde8a27ad, codex:019df40f-3672-73c0-8d5e-13abe5edd0f3 (+6)

9. Validate across >=3 seeds at full length with controls; trace fragile/star seeds individually and rule out run-to-run noise

loop · negative result

Single-seed wins are routinely reversed, so validate every experiment across at least three seeds (e.g. 42, 7, 99) at full 10k length: seed 7 repeatedly punished changes that improved 42 and 99 (a recruitment patch went 21.25->11.88 on seed 7), and several changes passed focused unit tests yet tanked seed 7. Keep a known-fragile regression-guard seed and run the full game on it before keeping any change; never ship on the strength of one good seed. Runs are nondeterministic enough that two can land on nearly identical trajectories (16.6792 vs 16.6794), so before attributing a regression to a mechanism, rerun the clean committed baseline to confirm reproducibility — an apparent handoff regression was actually the run falling onto the old baseline trajectory by tick 500 before the new mechanism could fire. Run noop and random partners as controls alongside the real partner to separate 'partner is weak' from 'partner actively sabotages' (cogs-vs-clips 6+2: noop 30.13 > random 21.11 > starter 9.94 proved the starter was worse than a do-nothing partner, so the loss was active resource interference, and apparent tournament 'aggression' is weak/uncoordinated partners, not a meaner opponent). Per-seed resilience varies enormously, so trace fragile seeds individually — seed 256 dropped -3% while seeds 1/512 dropped -72%/-53%, and the -8.1 average hid that loss concentrated in a few map geometries where extractors cluster by element near the hub. Investigate outliers before condemning/accepting a mechanism: a single-seed -15 catastrophe can be a bug (a retry/exclusion re-pick loop), but a whole class of changes regressing the SAME star seeds (three navigation tweaks all tanking seeds 1, 7, 1337) is an alarm about a load-bearing-but-fragile strategy — find the mechanism and test any concentration-reducing hypothesis at 10k. A small perturbation cascading into a large swing signals fragility, not a bad change (a one-cell miner path change cost -5.34 at 10k via a missing stuck-recovery mechanism). Run Coworld validation episodes sequentially (one seed at a time), since concurrent 8-agent matches make Docker timing noisy, and only advance if the current seed reveals no blocker. Stay alert to the classic 'looked-like-success' failures throughout: local-vs-live drift, stale rotating IDs or docs, over-reading a small batch, and position-based (rather than policy_version_id) score joins — each makes a run look like progress while the signal is an artifact. sources: codex:019e194d-3a17-7eb2-9df9-7bae24112862, codex:019e19a7-8605-7992-9b28-cb4d2ca9132e, codex:019e28e9-6b45-7c02-b4c7-bb9b0f44e022, ses_1fb28db30ffe6RjE6dq3HoVbzs (+5)

10. Run scored validation at the real tournament step_seconds, not an accelerated tiny value

loop · negative result

Run scored/behavior/diagnostic validation at the real tournament step_seconds (e.g. 0.02), not an accelerated tiny value, before changing movement/wrapper logic. A tiny step_seconds (e.g. 0.001) makes asynchronous action-observation lag far worse than tournament pacing and makes good, timely actions look like noops: a DEBUG run at 0.001 reported timely actions as last_action=noop, and the same 80-step stress produced 0-27 shifted-landmark duplicates per agent at 0.001 versus near-zero at 0.02. Re-run at the real rate before concluding from apparent noop results; only shrink step_seconds when explicitly stress-testing async lag. sources: archive/cogames_playground/bulbacog/designs/LOCALIZATION_ROOT_CAUSE_RE, codex:019e28a2-f81d-79b1-a7f7-5f6583c280f1

11. Diagnose regressions from replay artifacts and differential reproduction, not memory or argument

loop · ⚠ session-derived, unverified

When an experiment regresses, diagnose from saved replay artifacts (per-tick location, inventory, tags, talk history), not from memory or a design argument: reconstructing latest-talk/location/junction-ownership by tick distinguishes 'no eligible aligner was nearby' from 'an eligible aligner chose other work', and in one session the replay disproved the assumed cause (stdout is lossy and unordered - wait for the run to finish and parse the trace). Differential reproduction (flip exactly one variable) is the fastest confirmation of an externally-reported bug: paired short matches with ANTHROPIC_API_KEY unset vs a fake non-empty value isolated a hang to the key-gated LLM-guidance path without a real key. For a single-tactic fix, build a concrete numeric trace of the failure AND fix at one failing location, walking exact coordinates and distances through both old and new logic (old snap distance 27 then 31 at (745,134) triggers resync; new segment snap distance 0, progress 32px, lookahead 10px further down the same segment) so the fix is auditable without running the game. Close the loop with BOTH focused tests and a live-match action-mix check, then clean generated build artifacts: rebuild the shared library, run no-key AND fake-key 8-agent matches confirming clean exit (~45% non-noop, all manifests closed, no cross-bot LLM leakage), and remove stray Nim test executables so the patch stays source/docs/test only. sources: codex:019dfea2-76a2-7ba0-996a-9be68ceee04f, codex:019e00d7-d212-73a2-8b2e-569efc00f028, codex:019e191b-e5d6-7c43-b54c-8283e76cd26f, codex:019e194d-3a17-7eb2-9df9-7bae24112862

12. Separate observations from inferences and verify game mechanics and constants against installed engine source — engine is the tiebreaker

loop · negative result · ⚠ session-derived, unverified

During diagnosis, separate observations from inferences and forbid premature edits. Effective micro-loop: report a specific observed pathology with a causal hypothesis, explicitly forbid edits first ('Don't modify any code yet. Just look into these.'), cite exact ticks/slots/rooms, and re-verify game semantics against ground-truth source before building a fix. Do not invent mechanics to explain trace anomalies — an agent attributed a 20-second localization loss to a nonexistent 'task minigame UI overlay covering the map' and built a whole root-cause story and fix before a human caught no such mechanic exists. Verify game version and feature presence against actual source plus replay hashes before trusting human recall: a recalled 'they removed crew_tasks_remaining and added meeting-caller indication' was half wrong — git log -S showed no removal commit, the counter still renders in the live player stream (server.nim), and replays hash-validated against current master HEAD. Before testing any threshold or tuning parameter, verify the actual game constants (cargo, extractors, roles, mission modules) in the INSTALLED game source, not docs, memory, or old packages — the live game can have drifted. Audit a player's design/strategy doc by reading the GAME's authoritative documents first (rulebook, game API, state schema) and cross-reference every quantitative claim against ground truth; the most damaging defects are 'false assumptions' the design treats as fact but the rules contradict. Resolve open questions from engine source (sim.ts / perception), not the doc — when auditor and author disagree, or when an empirical run-episode/play execution contradicts a claim read from source, engine source / the run is the tiebreaker (it overturned doc claims about menu-blocked-during-summit, minimap shadow logic, perception-panel handling; correct the doc from the run, not the run from the source). Time/budget math is high-yield: a doc claimed an action cycle was '8-10s -> 1-2/round' but its sub-steps summed to 12-18s -> ~1/round, miscalibrating the strategy layer. sources: claude-code:0919285d-1374-4827-b596-6e7107d95526, claude-code:0e7b14ca-ffd5-4137-9456-47485d3c6f87, claude-code:3c867a71-3057-4e87-a149-4185e82ee728, claude-code:67ff1fe3-48c0-4738-b7fb-8b994e567434 (+2)

13. Change exactly one component per cycle and A/B it, revert on regression, and win first before generalizing

loop · negative result

Run the improvement loop as: analyze exactly ONE weakness, propose a fix with file:line plus risk, implement ONE change scoped to a single hypothesis (surgical — tweak constants, add guard clauses, adjust thresholds; never rewrite whole functions or change class APIs; keep edits to 1-3 targeted changes in the named files, and keep tunable knobs in a config layer separate from logic), eval against baseline, then submit if improved or REVERT if regressed. The two non-negotiable guardrails are exactly one weakness per analysis cycle and 'revert on regression, no exceptions', because one change per cycle keeps attribution clean and a broad rewrite confounds which edit caused which effect. The ship bar is A/B-observable behavioral change, not soundness: a semantically-correct refinement producing byte-identical scores is a rejection (keep its general plumbing for future hypotheses), and even a 'pure' bug fix can regress — separating A* transit from interaction dropped max-stuck from ~5875 to 11 ticks but regressed tournament mean by -0.46 because reliably reaching every target doubled deaths through enemy territory. Sequence overall effort as 'win first, generalize second': push a single player to a sustained league placement (a rolling-N-day median rank, not one lucky run) before pulling effort into platform/tooling generalization. Corollary: perception completeness is not the goal, competitive policy is — spend perception effort only on what the current strategy needs and ship better bots before closing un-fixtured perception edge cases. sources: claude-code:2426100f-257d-4100-ba93-2097175a999c, claude-code:7db8d397-c6dc-47b6-9b95-8f90675dd1b8, claude-code:fa645b7b-4fa0-4b55-992e-b273ee703391, optimizers/skills/continuous-optimization/SKILL.md (+9)

14. Pin exact test conditions to every finding and keep a per-version/per-submission log keyed on the policy-version UUID

loop

Capture exact test conditions inline with any policy critique or finding — player count, imposter count, match length, server seed, kill-cooldown ticks, and whether LLM guidance was active — because a finding is only reproducible and comparable when its conditions are pinned alongside it. Keep a per-version log keyed on (name, version) with the policy-version UUID as the canonical id, one row per submission/upload: date, version label, commit, env/flags, league id, submission id, resulting membership ids, how it was A/B'd (experience-request IDs per arm), hard results including negatives, and a one-line behavior-delta description — so each league result is attributable to a specific behavior change and score regressions correlate to their cause (for re-uploaded arms note the image is identical and only the secret env differs). There is no coworld versions command, so reconcile against the live list; keep the log under that player's lab dir, not the lab root; and embed a timestamp in each submission name so successive iterations are distinguishable. Submission/tournament bookkeeping (committed tournament JSON snapshots, the current-version submission, dirty-tree submissions) vanishes between sessions unless written down, so log it as explicit caveats in the lab notebook. sources: claude-code:307f1069-21fb-469e-bcd2-8a07f2dba0d8, claude-code:91f456a7-41e8-4a89-98e9-44e379160b15, codex:019df3f0-db41-7561-8184-8832002dba41, personal_labs/crewrift_lab/crewrift/crewborg/version_log.md (+2)

15. Commit research-state first and treat trace artifacts as git-ignored regenerable outputs

loop · see related: S007 (other tier)

'Audit checklist checked off' is not 'cleanly landed': a research-audit file can mark hygiene items complete while the worktree is still dirty, so commit the baseline research-state (audit/status/notebook edits) FIRST so each subsequent experiment gets its own clean, separable commit. Trace JSON files are deterministic outputs of a (policy, seed, steps) tuple, so gitignore them and reproduce on demand rather than committing them — they are large (~100 MB) and reproducible, so verify they are ignored before staging or a git add will swallow them; keep generated baseline/comparison artifacts out of the test fixture tree and repo root, in a clearly named untracked directory alongside the plan. Before dumping a large (e.g. 128KB) working-memory snapshot per tick into the replay, empirically verify with a probe policy whether the rollout truncates large _infos, or the snapshot may be silently lost. sources: codex:019df40f-3672-73c0-8d5e-13abe5edd0f3, archive/cogames_playground/alpha_cog/experiments/traces/README.md, opencode:ses_21e25877effeMecaawB2dUys7v, opencode:ses_223c540d9ffe8nF6kcU1RXf0wp

16. Validate a destructive recovery procedure against the live failure, and tear down leftover PIDs between games

loop · negative result · ⚠ session-derived, unverified

Before depending on a documented recovery procedure, validate it against the live failure: confirm its exact preconditions are actually present and dry-check what a destructive step would delete before letting it run, rather than firing the destructive step blind. In batch runs, a leftover background process from a prior run can silently kill the next game's server (one game died 'server died early (rc=0)' / 'killed by leftover process'), so tear down all agent and server PIDs and free the port before each game, and record per-game whether the game actually reached GameOver. sources: claude-code:91060045-312c-4787-9819-a1421d228d61, claude-code:aae25940-ade3-41bc-b817-517dc86ccebf

17. Chase cross-cutting actuation/latency bugs even when their per-game score looks small

loop · ⚠ session-derived, unverified

Chase a cross-cutting actuation/latency bug even when its direct per-game score impact looks small, because a systemic cost taxes every game. Example: a player sitting at spawn far longer than other agents at game start turned out to be rebuilding its nav graph and occupancy substrate from scratch on the first streamed scene every run — a real actuation/perception-initialization bug, not a strategy or voting issue. sources: claude-code:f70c9801-7ee7-499b-97f9-4fd0848a6b8e

18. State explicitly what was NOT checked: which scenarios were live-validated vs only unit-tested, plus uncertainties, treating doc 'next steps' as stale

loop · ⚠ session-derived, unverified

Be honest about validation coverage gaps in a handoff: state which scenarios were actually exercised in a live run versus only unit-tested (vent/teleport traversal was unit-tested and wired but validated live only in a crewmate scenario, so an imposter-specific live trace was still owed and the handoff said so). If the environment blocks a live run a smoke harness exists for (e.g. scripts/orpheus_live_test.py), explicitly state validation stopped at the test suite and name the remaining risk; never imply a live run that did not happen. A useful status report covers four things plus an explicit not-checked list: (1) design state per role (does design.md mark each role implemented end-to-end), (2) focused validation (the player's pytest suite ~235 passing plus ruff), (3) git cleanliness vs origin/main, and (4) an explicit statement of what was NOT checked (live hosted league performance, recent Observatory episodes) — local tests passing does not mean live league performance was verified. Record genuine uncertainties as explicit 'uncertainty' notes in rulebook/design docs rather than asserting a guessed rule (flag whether the Cogs-v-Clips clips initial-seed event fires at tick 1, how territory energy stacking behaves). Treat 'next steps' / 'minimal next changes' lists in design docs as historical and likely stale — their items are often already tested and dated to an early landing, so cross-check against the live experiment index before acting. sources: codex:019dfe75-7dd6-7dd3-90af-84bb28fda415, codex:019e04e9-dfe5-77e3-9d43-4c84876cc530, codex:019e08cf-4f50-7ef1-bc11-82b8d92354b8, codex:019e9354-431e-7ae1-95a8-358bb4cf53de (+2)

19. A submission, a 200, or a green test suite is not verification — require the real end state

loop · ⚠ session-derived, unverified

Do not declare victory on an act or proxy signal: an experience-request 200 with a generated ID (IDs that never became readable/dispatched), a promotion/qualification (Qualifiers rank 1 still left Wood rank 1 unmet because the round was still running), a passing pytest/ruff suite, or a single-agent smoke run can all look like success while the real outcome failed. Require a completed readable run with a real job; prove the end state against current completed leaderboard results; validate players in the real multi-client match configuration; reproduce the original failing scenario end-to-end. When verifying a specific policy version, target its policy-version ID — a 'latest episodes across all versions' query sorted by created_at surfaces the newest-by-timestamp version (v6) instead of the one you want (v15). Tests that were not failing prove nothing; distinguish 'structurally correct' from 'actually unblocks the behavior.' sources: codex:019e191b-e5d6-7c43-b54c-8283e76cd26f, codex:019e1940-5a75-7f51-b057-63f894ddb061, codex:019e4c28-d516-7ca0-b145-153c152b4f43, codex:019e9354-431e-7ae1-95a8-358bb4cf53de (+7)

20. After a fix, confirm via traces it actually FIRED, and verify with before/after outcome metrics not 'no crash'

loop · negative result · ⚠ session-derived, unverified

A fix that compiles and passes unit tests can never trigger live: a rewritten histogram teammate-detection produced zero imposters_detected events in the next match (self-color unknown at the interstitial / max-colors guard / sub-threshold counts), so the bot fell back to inference. The strongest behavior-fix confirmation is reproducing the ABSENCE of the bad event in a live trace (zero body_newly_in_view_flee firings across a focused match with confirmed kills and body re-sightings) using a known reproduction shape (seed 100 / cooldown 48). Verify with productivity/outcome deltas from real traces, not raw counts: task completions 2->9, and scan for long consecutive RUNS of the error marker (old bug = 60+ stuck ticks) because 265/4029 (~6.5%) transient occurrences still passed. For a transport/timeout fix you needn't run a full match — cap maxTicks (e.g. 2500) to push past the known failure window (old ~40s / tick-~905 drop) and scan ALL player logs for absence of close/keepalive/traceback past that point. sources: codex:019e191b-e5d6-7c43-b54c-8283e76cd26f, codex:019e2819-d56b-72f3-9349-60fc965b6ed2, ses_1f7b6dc30ffeDmQtlUZI25HAjh, ses_1ff325156ffei2UiDygaVOZo5O (+1)

21. Localize the failing layer with cheap probes, per-role metrics, and per-tick traces before touching strategy code

loop · ⚠ session-derived, unverified

Diagnose each pipeline layer independently against the replay; do not attribute a bad outcome to the layer you just changed (a new 5-phase hunting logic produced clean transitions but 0 kills because a separate layer, teammate color detection, was broken; a strategically-correct LLM still drew with 0 rendezvous because win-condition mechanics were broken) — diagnose win-condition/action-execution separately from decision quality. Use cheap targeted probes over re-running noisy matches (a direct TypeScript engine probe confirmed the role offer/accept menu worked, isolating the bug to agent-side navigation). Quantify before reading code: a per-role noop fraction (69% imposter vs ~29% crewmate) localizes idling. Verify a root cause against a per-tick decision trace before committing to a fix (static reasoning blamed action-garbling for a 100%-noop bug, but the trace showed correct idling on frames 1-120, role detection at tick ~121, then stuck because localization never succeeded) — sequence a diagnostic step before the fix for ambiguous causes, implement directly only for unambiguous ones. Diagnose cascades with a tick-by-tick divergence trace (a miner's step at tick 1739 shifted positions, changed the aligner's 13x13 window at tick 2018, costing 10 cycles x ~0.53 = the exact 5.3 regression), and trace one dependency chain rather than fixing symptoms (no-role -> no-kills -> no-bodies -> no-reports -> no-meetings). sources: codex:019e131a-1bb0-7400-b54a-f208202b291d, codex:019e132d-8773-7503-86cd-a77b0a8d21eb, codex:019e181f-6289-7332-9837-05f0e9cadded, ses_1f7b6dc30ffeDmQtlUZI25HAjh (+4)

22. Verify the actual cause against history and isolate one variable — don't blame the change you just made

loop · negative result · ⚠ session-derived, unverified

When a previously-passing upload suddenly needs a new flag, verify against git history rather than assuming the certifier/runner changed (the real break was a game-side default flip: crewrift commit cc27a51 made notsus reconnect-by-default instead of exiting). To isolate a suspected hosted-orchestration variable, resubmit the identical image with that one variable removed while holding image, entrypoint, LLM defaults, and runner contract fixed (testing whether --use-bedrock switched pods onto a failing service account). When a new feature appears to cause a failure, prove it before fixing: radar exclusion showed 0/8 tasks, but the cause was a pre-existing lifecycle bug (server accepted a task at ~72 of an 84-tick hold; an unshielded 2-frame icon-miss counter fired at ~tick 78 when the icon legitimately vanished). A bug only in full self-play but not mixed matches is a population/composition effect — confirm both run paths share infrastructure (same find_server_binary() helper) to rule out the binary before chasing thread-safety theories. sources: codex:019e2886-d576-7770-be1b-6d92ca97761e, codex:019ea9fe-29e5-7180-b111-57e3039314a2, ses_1fc4cec1bffe16bZw4eHP3SDBL, ses_21a8f9dedffekGKNrfPfGruC81

23. Window every per-run metric to the live-game segment and label server-authoritative vs bot-local stats

loop · ⚠ session-derived, unverified

When a metric looks impossible (51 tasks completed against a hard ceiling of 48), distrust the aggregate before the game: the count was inflated by trace events summed across the whole fixed-duration run, including events fired after game-over while the server kept serving frames — cut every per-run metric at the first game-over signal. Don't trust a single aggregate like max_steps_without_motion to confirm a specific-bug fix (it mixes in unrelated endgame idle); inspect the replay around the exact failure window and distinguish idle causes (full-heart stationary near hub dropped to <=1 tick vs zero-heart agents waiting on restock). Always label whether a stat is server-authoritative or a bot-local heuristic: a bot's local task_completed (held station N ticks until the icon vanished) is not the server's counter, and the server can accept the final task and trigger game-over before the bot emits its local completion. sources: codex:019e132d-8773-7503-86cd-a77b0a8d21eb, codex:019e1355-fd7e-77b2-8af2-3f93c0ae623a

24. Treat seeds skeptically — verify they reach the subsystem you need and actually vary the games

loop · negative result · ⚠ session-derived, unverified

Do not assume N episodes with the same roster get N different seeds: a high-impact backend bug computes a distinct per-child seed (sha256(parent_id:job_index) mod 2^31) and stores it on the request row, but the dispatch path builds the job payload only from game_config + policy_version_ids and never injects it, so every episode runs the default game_config.seed — N identical games, wasted compute. Trace the seed into the embedded seed in completed replay headers; a stored seed field is cosmetic. A server --seed may not uniformly control map, policy RNG, AND role assignment — a single agent in slot 0 may never hit the crewmate branch, so use a multi-agent harness, and treat any seed-pinned slot-to-role mapping as a debugging convenience, never a game invariant. Byte-identical decision traces across different server seeds are a red flag for hidden global state (a global mutable sequence holding policy instances in the FFI layer, a forced-role flag that silently does nothing, or a purely reactive policy) — use a cross-run diff as a cheap nondeterminism-leak check. Tournament-runner seed-forwarding fixes don't affect local scripts that build their own env config with explicit seed=, so set seeds explicitly locally and add a master-seed flag with a sentinel default (initBot(mapPath, masterSeed=-1)) plus a --seed: CLI flag to enable reproducible parity testing. sources: codex:019ea9af-7020-7550-b427-d3478fed2057, ses_201bc0c24ffe4uYJuw52Gs42LG, ses_2051bd225ffefNc9y2aOEEFjHW, ses_20ba27dfcffetaDrq3FK2cz523 (+2)

25. Re-query live coworld/league/policy IDs, verify each experience request on creation, and stagger eval batches before POSTing

loop

Coworld, policy, and league IDs are time-sensitive and memory notes go stale (observed drift: crewrift 0.1.31 vs canonical 0.1.32; another resolved 0.1.35, league Crewrift Daily, crewborg v4): re-query production for the canonical coworld id, current league, and policy version immediately before POSTing, since the endpoint takes a raw coworld_id and the league lookup is the source of truth. Standings and top-N membership move mid-pull (a #7 player swapped out during collection), so refresh right before POSTing. Resolve opponents by joining leaderboard rows to active memberships on player_id (prefer substatus=='champion', else newest active by created_at), not by copying the CLI table. Start with num_episodes=1, confirm a job_id appears on child episodes, then stagger N-episode requests one at a time (10x100 run as sequential single batches, not 1000 queued) to avoid overloading the rate-limited backend; if polling stalls ('15 running, 85 pending' unchanged), inspect job/episode state to distinguish slow games from a stuck scheduler. Immediately after creating an experience request, read it back and verify episode_count equals the requested count, child episode rows exist, participant order is correct, slot 0 is the requester policy, and opponents resolve to the expected policy names/versions. Do not report a benchmark as 'done' just because the request was created; track and report distinct states separately: request accepted, jobs submitted, games running, games completed, and failures. sources: codex:019e746d-062c-78d3-a0ed-ef2fcb892baf, codex:019e8afb-efcd-7ca0-820f-0320501ef2fa, codex:019e8fc7-ebad-7b00-bc95-df1727641ce0, codex:019e93a0-50c3-71a0-a0dd-c1935af4add0 (+3)

26. Stage mechanics before strategy: build a policy bottom-up in numbered stages, each with a measurable acceptance gate, shipping the smallest version that beats the bar

loop

Never debug strategy and mechanics simultaneously: prove a mechanic with a trivial deterministic target first (tell bots to vote a fixed offset, bot 2 votes bot 3, to confirm the parse-and-aim pipeline can hit a chosen target), then layer strategy on top. Make scripted triggers deterministic about action timing — be explicit about input edges (fresh button press vs held input) and step ordering; trigger an emergency meeting with a fresh button press at spawn, not by relying on prior movement. Build a game-playing policy bottom-up: first a valid image that connects and sends valid input, then reliable perception export plus memory recording, and only THEN a single LLM call — never start with clever LLM strategy. Build in explicit numbered stages with a local-game validation check after each (Stage 0 = types/state/knowledge scaffolding verified by import-only checks; later stages add decision logic), ending each stage with a focused import check of exactly the new modules, and give each phase explicit done-criteria depending only on capabilities proven by earlier ones. Concrete example (coborg): P0 noop completes a 120s coworld run with stderr traces; P1 perception parity green vs Nim with under 8 ms per tick; P2 crewmate completes >=3 tasks in 120s; P3 full meeting cycle with a legal vote; P4 imposter scores >=1 kill on imposter-pinned seeds 50 and 100. Sequence dependencies so role strategy does not depend on deception/leadership before the generic whisper protocol, knowledge updates, and evaluator parameter contracts work; cap a deterministic baseline's scope (rule-based imposter) and defer LLM strategy to a follow-on plan so the baseline ships before model-driven complexity. STOP at the first stage that wins enough and advance only when the measured metric (time-to-reacquire, kills, cell accuracy) justifies it — ship a deliberately scoped first version validating the core mechanism before adding state transitions (bulbacog v2.0 cut dead-code scouts, deferred scrambler/raids, aimed only to beat the 2.94-per-cog ceiling at acceptance >=3.5), but keep future-intended modes in the tree when reintroduction is additive. When recovering a deleted 'working' module, restore it verbatim rather than refactoring — a behaviorally-equivalent rewrite risks silent strategy regressions import checks won't catch; refactor only on explicit request. sources: codex:019e05ca-d8e8-7d02-adf9-43354a64543b, codex:019e06a0-5754-74e0-baf2-771853591aaa, codex:019e132d-8773-7503-86cd-a77b0a8d21eb, bitworld/among_them/players/SMART_BOT_GUIDE.md (+5)

27. Follow the fixed pre-ship protocol, smoke-test candidates first, and keep league submission a deliberate manual gate

loop · ⚠ session-derived, unverified · tool: ship_to_league.py

Pre-ship protocol for a configurable Coworld policy, in order: (1) regression check confirming all-flags-off byte-matches the starter; (2) A/B at 12 seeds x >=2000 steps; (3) chronological historical-roster tournament confirming no regression vs any prior ship; (4) after editing decide.py or pursue_target.py, a 3-seed spot-check of recent versions. A fired --chronological tournament gate (exits non-zero when any older roster entry beats the candidate on mean OR wins more seeds than it loses) means 'do not ship until investigated,' NOT 're-run until it passes' — read the regression report; a clean A/B does not override it. Smoke-test a candidate before a multi-seed A/B with cheap signals in increasing cost: instantiate the policy class against the installed framework, run the player's unit tests, dry-parse each helper script's args/imports, then run a single episode to catch load/import/shape crashes. Keep league submission a deliberate MANUAL gate even in a 'commit after every experiment' workflow via a wrapper exposing ship/status/history subcommands requiring an explicit version (never auto-ship). The full loop: download tournament replays -> trace.py diagnoses -> ab.py A/B tests -> ship_to_league.py ships -> repeat; tournament replays are the true experiments, local self-play is exploratory. sources: ses_1fa6f7b71ffeMqiFWPAD235jUs, ses_1ff344748ffeIooJBfFGRt6YG0, ses_201bc0c24ffe4uYJuw52Gs42LG, ses_2095d36efffemO4jJhy7HRYYxD (+3)

28. Validate at the full 10000-step horizon with canonical seed sets, and don't react to leaderboard rank before ~20 matches

loop · ⚠ session-derived, unverified

Always validate hub-dynamics, gathering, navigation, and safety hypotheses at the full competition horizon (10000 steps), not just 2000: multiple flags reverse sign between 2k and 10k (saturation-fallback multiplier, streak-suppression), and some are byte-identical across all 12 seeds at 2k only because their trigger (enemy-territory extent, path contention, deep-territory energy bottleneck) is too rare in 2000 steps — a flag inert at 2k is NOT evidence it is useless. Energy-scarcity ideas need scenarios that create the scarcity: A* navigation gives friendly-territory agents effectively +100 energy/tick, so the night bottleneck only manifests with deep-territory aligners on long runs. Use the project's canonical seed sets (using range(1,50) silently invalidates comparison): the 50-seed set is random.seed(2026) then 50 random.randint(1,99999) calls (NOT random.sample); the 12-seed quick set is 1,7,13,42,99,123,256,512,777,1024,1337,9999. Promote reproducible 'character' seeds to a named diagnostic-seed probe list. The first reliable leaderboard signal arrives only after ~20 matches — do not react to rank before then. sources: ses_1ff344748ffeIooJBfFGRt6YG0, ses_200493314ffeuU44LQ1zuZaduk, ses_2042c1ff1ffefETImBU5n4D43k, ses_21a4b4fa0ffe6EhdZLacMU1JE4 (+1)

29. Verify per-seed-trace outcomes over multiple seeds, and verify the pipeline can absorb a local fix before investing

loop · ⚠ session-derived, unverified

When verifying a player end-to-end, run multiple seeds at decision-trace level and tabulate per-seed outcomes (detected role, objectives completed, stuck?, noop%) — a single run hides intermittent failures (bugs appeared on ~50% of seeds, so a one-shot 'it works' would have missed them). Before investing in a local improvement, verify the rest of the pipeline can actually consume it: a pre-gear agent-behavior fix increased throughput but the downstream pipeline couldn't consume it, so it stayed low priority. Watch for the NEXT bottleneck a fix exposes rather than declaring done — fixing navigation revealed the bot then holds the task action indefinitely (no task-completion detection); record the newly-exposed issue as the new highest-priority item. sources: ses_20ba27dfcffetaDrq3FK2cz523, ses_219f1c8cdffe3wZlz1iudO5c4O, ses_21a708a4dffeBXpeHY15fnGABP

30. Diagnose with replays and a corpus study before writing experiment code

loop

Up-front investigation -- replay analysis, metric comparison, per-split breakdown, or a corpus study over existing replay data -- frequently identifies the mechanism and solution directly and saves hours of undirected experimentation. A Phase-0 'where and how often' question often needs zero policy changes because expanded replay JSONL already provides caller-attributed meetings, per-slot position samples every 24 ticks, roles, and map geometry. Gate any speculative niche behavior on such a study (frequency, timing, interceptability) and decide build-or-drop on the numbers. sources: alpha_cog/AGENTS.md, personal_labs/crewrift_lab/lessons_archive/TENTATIVE_LESSONS-20260613-, personal_labs/crewrift_lab/crewrift/crewborg/docs/designs/button-runne

31. Make in-loop policy code deterministic with tick-time and per-consumer seeded RNG substreams

loop

In-game-loop policy code must read time only as an integer tick (never time.time/monotonic/datetime.now), read no env vars, touch no filesystem outside its own tree, and sort sets before iterating when order affects the action (set order is PYTHONHASHSEED-dependent across processes). Draw randomness only from architecture-provided RNG (never import random/secrets/numpy.random/os.urandom), and give each concern (chat/task/follow/tie-break/role-flip) its own substream derived from a master seed plus stable inputs, e.g. random.Random(f"bulbacog-v2:{salt}:{agent_id}:{tick}"); this took modulabot to 6281/6281 (100%) self-consistency and makes the probability sequence reproducible for replay debugging. sources: archive/cogames_playground/alpha_cog/docs/agent-architecture.md, bitworld/among_them/players/mod_talks/DESIGN.md, bitworld/among_them/players/mod_talks/LLM_VOTING.md, bitworld/among_them/players/modulabot/DESIGN.md (+1)

32. Identify runs by a reproducibility tuple, pass an explicit seed, and quarantine dirty-tree runs

loop

Make a scripted-policy run reproducible by the tuple (seed, mission, variants, code-git-sha, initial-memory) and freeze the skill library / stage programs for the run (the optimizer edits only between runs). Always pass an explicit seed and name the exact reproducing command, because a simulator that picks a non-deterministic default seed when --seed is omitted makes two identical-looking commands diverge. A dirty git tree is NOT reproducible: record the dirty state in the trace and discard such runs for accept/reject decisions. sources: archive/cogames_playground/alpha_cog/docs/agent-architecture.md, archive/cogames_playground/alpha_cog/experiments/notebook/protocol.md

33. Test small logic as pure helpers, validate merges with per-feature suites, and isolate pre-existing hangs

loop · negative result · ⚠ session-derived, unverified

For a small heuristic detector, extract a pure helper with unit tests rather than extending a slow/hanging integration script (bandwagon detection became a pure tallyBandwagon helper with 5 scenario tests instead of extending the hanging trace_smoke.sh), which makes threshold/window edge cases like strict > vs >= testable in isolation. Validate a perception/policy merge with the player's own focused per-feature suites (speaker-attribution, vote-bandwagon, trace-smoke) PLUS the game's full suite, not a compile check. A long-hanging test that ALSO hangs on a clean master checkout is a pre-existing infra issue, not your regression: trace_smoke.sh hung on master while parity passed on 500 and 5000 frames, so the change shipped on the working parity/validate suites. sources: opencode:ses_21f5b8db7ffe8xs4COwtdx913c, opencode:ses_21ece5489ffeFZNEP6Ehx80GI1

34. Make tuning constants configurable, calibrate them to the server empirically, start from theory, report perf as budget percentage, and log untuned thresholds as open

loop

Expose latency-sensitive tuning constants (a vote-listen-ticks window) as env vars or config rather than hardcoding them in a compiled policy, since a hardcoded constant forces edit-and-rebuild per experiment. Calibrate timer constants to the server: a conservative meeting-length estimate (e.g. 600 ticks ~25s) matches the local server, but tournament servers may differ, so prefer empirically detecting meeting duration from past meetings over fixed fallback-timing constants that risk firing too early or too late. Start window/buffer/threshold params from theoretically motivated values and only sweep if the initial choice proves inert — an A/B-validated hyperparameter often has a Goldilocks point (sweeping stuck-detection window 60-180 ticks and threshold 1-3 found window 120 / threshold 1 fires on genuinely stuck agents without catching transient navigation hiccups). Report optimization results against the tick budget percentage ('BotCore.step p50 21% -> 6.6% of the 42 ms budget, p99 28% -> 13%') not raw speedup, to show the real margin and remaining cliff. Record any magic threshold introduced to fix a behavior (a 32px teleport re-lock) but never empirically tuned as an OPEN item, not a validated constant — unverified constants are latent regression risks. sources: opencode:ses_21e7dd4d4ffewtISu38YXrsOgs, opencode:ses_21e7f66a8ffeNFrL2URK8EWAn7, bitworld/among_them/players/mod_talks/DESIGN.md, personal_cogs/among_them/guided_bot/MEETING_DESIGN.md (+1)

35. Fix the dead-reckoning action-tracking source of truth before tuning startup behavior

loop

Fix engine-applied action tracking BEFORE tuning collision-prone startup: a changing action sequence corrupts dead-reckoned position faster than a stable one, and startup convergence sending all agents to the same hub amplifies action skew. Do not overfit staging/wait-point logic before action tracking is corrected. sources: archive/cogames_playground/bulbacog/designs/LOCALIZATION_ROOT_CAUSE_RE

36. Match iteration cost to the loop: run network-free unit tests as the fast loop, then validate runtime behavior by actually running the policy against the exact tournament runtime

loop

Since the coworld CLI migration there is no in-process cogames play -p class=...: every policy change needs a docker build before coworld play/run-episode, so keep the in-process unit suite (uv run pytest, ~0.5s) as the fast loop and reserve the ~30-60s rebuild plus full play for behavior checks (a scripted policy smoke runs to completion via uv run cogames play -m arena -p class=scripted_alpha.StarterPolicy -r log -s 30). Run player unit tests with no network so they force the deterministic (non-LLM) decision path, keeping them fast and reproducible; exercise the LLM path only in explicit local episodes with platform Bedrock creds. After implementation changes, run the narrowest useful static/unit checks without reviving deleted run paths, then validate end-to-end through uv run coworld ..., inspecting episode logs and stderr JSONL traces; distinguish a manual smoke harness (launches a server, starts the agent, optionally adds fillers, verifies expected log transitions) from the pytest suite and document it as such. To parity-check tournament behavior, run a local game through the EXACT tournament-worker runtime (metta's bitworld_runner.run_bitworld_episode driving the cogames-style Python wrapper with its concurrent ThreadPoolExecutor); for most non-tournament local runs prefer the lighter direct-CLI path. When a scripted-policy score is far below expectation, first run the scripted policy itself end-to-end and read raw outcome metrics (action.move.failed rate, energy.gained, max_steps_without_motion) before blaming the learned policy — if the reference also fails, the bug is in the environment, not training. Auto-stamp the run config (LLM provider, model, persuade/disabled flags) into the run manifest's harness metadata at launch so downstream analysis groups runs by config without manual bookkeeping. sources: archive/cogames_playground/bulbacog/COGS_V_CLIPS.md, bitworld/among_them/players/mod_talks/scripts/launch_mod_talks_llm_loc, opencode:ses_223dc7136ffeCr4ZslnR426sVG, players_2/players/proxywar/warlord/README.md (+4)

37. Promote only on a promote gate decision; on reject/weak_evidence go back to diagnosis

loop

Promote a candidate over the champion ONLY when the eval gate decision is promote; never promote on weak_evidence or reject — the gate is the optimization loop's objective function. On a reject or weak_evidence decision, record an anti_pattern memory and a refuted/inconclusive experiment verdict, then return to diagnosis with a different hypothesis. If weak_evidence is due to low completion, increase episodesPerRun or episode count and re-run rather than treating it as a decision; log every eval outcome with experiment_log so the promote/reject decision is auditable. sources: optimizers/CLAUDE.md, optimizers/skills/hermes-operator/SKILL.md, optimizers/skills/improve-policy/SKILL.md, optimizers/skills/policy-eval-gate/SKILL.md

38. Gate, cancel, and roll back: bound optimizer cost with safety primitives

loop · tool: guardrail_check

Before any expensive or mutating optimizer action (episode_launch, episode_run_sync, eval_enqueue, eval_launch_campaign, policy_promote, policy_submit_to_league, skill edits), run the cheap read-only guardrail_check (and/or budget_check) with the matching action; if it returns allowed:false, STOP and report the blocking reasons (over budget, over episode ceiling, too many running) rather than launching, and never retry to get around the ceiling. Use budget_check (overBudget, remainingCents) and cost_report for a direct budget read. Prefer cancelling wasteful work (episode_cancel, eval_cancel, opt_cancel) over running to completion; if a promotion turns out bad, rollback (demote champion, promote known-good). Use loop_health on a slower cadence than the main loop for a one-call snapshot of watchdog reconciliation, concurrency, and recent runs to find stuck work. sources: optimizers/README.md, optimizers/.cursor/skills/mcp-optimizer-operator/SKILL.md, optimizers/docs/optimizer-primitives.md, optimizers/skills/optimizer-safety/SKILL.md (+2)

39. Verify the intermediate metric maps to score, and mine positive outliers for mechanism

loop

Do not optimize an obvious intermediate metric without checking it actually maps to score/win; counterintuitive correlations (e.g. dying more while scoring more) are signal, not noise. Mine positive outliers ("we got lucky / did unusuall

Truncated - read the full file at https://github.com/Metta-AI/coworld/blob/0ea5cba6d8ade6937942898ee76b5240bb84e230/templates/optimizers/252026/jboggs-crewrift-player-optimizer/optimization-loop-specific/skills/experiment-discipline/SKILL.md.