Imported from Metta-AI/coworld (
templates/optimizers/252026/jboggs-crewrift-player-optimizer/optimization-loop-specific/skills/perception-decoding/SKILL.md). Install upstream withnpx skills add Metta-AI/coworld --skill perception-decoding. Copyright stays with the author.
Perception & state decoding — recipes (loop tier)
On-demand recipes (29). Trigger→action heuristics; pull the relevant one when its situation arises.
1. Split a scripted game player into perception / belief / policy layers with a stateless perception core
loop
Structure a scripted policy as thread-safe layers and enforce the invariant that raw frame bytes/pixels NEVER leave perception: perception is a pure stateless parser parse_frame(raw)->FramePerception (one frame in, typed symbolic facts out, no memory, reports OBSERVATIONS not interpretations -- emit 'a color-3 dot on the minimap', not 'Hades is nearby'); belief is a separate step merging facts into persistent state (fast loop writes, slow loop reads under a lock); selection/policy reads belief and emits action mask + chat, consuming typed intents only. Keep raw frame buffers, token streams, and pixel atlases OUT of belief. Keep the raw scan layer (actors, bodies, ghosts, task icons, radar dots, OCR) SEPARATE from the task/belief state machine: raw scans emit only match records while icon-to-task assignment, radar projection, checkout latching, and target commitment live in the policy. Percepts must be typed, bounded, ADDITIVE (new detector fields add without changing existing meanings; to structure new info like whisper messages add a typed belief field plus parsing, NOT logic inside perception; probabilistic detectors preserve uncertainty and hard-vs-weak provenance). Build perception->belief as one pure mutate-in-place apply(belief, perception, previous_view) that threads the prior view explicitly so transition-triggered logic (non-Lobby->Lobby reset) is testable with one canned frame; add new perception capability inside that hook. Honor hard ordering dependencies (actor and task scans need camera localization first; task-icon scan after actor scan; ignore-mask exclusions stamped in a fixed documented order so later scans don't re-detect earlier sprites). Pick ONE canonical perception channel and enumerate documented exceptions rather than mixing sources ad hoc. This split is what lets perception be unit-tested against captured fixtures independently of the agent. Distinguish a per-frame View from a high-level GamePhase and map many views to one phase (WAITING_ENTRY/WHISPER/GLOBAL_CHAT/INFO_SCREEN -> PLAYING); split a view into its own type only when policy semantics differ. Treat the belief->LLM snapshot as its own versionable module that converts screen coords to WORLD/room space and derives non-first-class events (body seen, meeting started, role revealed) by edge-detection across frames. For an LLM-driven player keep moment-to-moment action selection on the deterministic symbolic reflex+mode path and gate the LLM to social/meeting/voting phases (toggled e.g. by GUIDED_BOT_LLM_DISABLE), emitting a guidance_suppressed trace so the symbolic-vs-LLM boundary is observable. CAVEAT: high perception/localization quality does NOT imply good behavior -- localization can be 96-97% accurate while the policy is useless. sources: players_checkouts/players/archive/players/among_them/coborg/DESIGN.md, players_checkouts/players/users/james/personal_cogs/persephone/README., DESIGN_perception.md (Persephone perception module design), auggie:e251085a-b35d-4171-8dbd-d006b8bc2bd9 (+17)
2. Order memory subsystems spatial->declarative->temporal with clear write-ownership
loop · ⚠ session-derived, unverified
Structure a localizing game agent's multi-subsystem memory layer (spatial, declarative, temporal) with clear write-ownership and update order. Each tick update spatial first (it establishes the agent's absolute position via localization), declarative second, temporal last, since everything keys off the spatial position estimate. Let ONLY the decision layer write declarative memory while spatial and temporal are populated from observations, so perception layers are not polluted by belief/strategy state. Implement temporal memory as keyed ring buffers with per-key capacity and a configurable sampling policy (every_tick / every_n / on_change) rather than logging every signal every tick; on_change sampling is what makes cheap stuck-detection and trend queries possible. Decide the natural ownership scope of each belief field before adding it: place cross-cutting transient timers (e.g. a grace-period counter, a previous-view-transition marker) on the per-agent state object, not a per-sub-entity object, or one entity's timer resets another's. sources: codex:019e05eb-dbd6-76f0-8bf4-3ef821cb7cc6, codex:019e0850-2833-7773-a5e4-80ee662745c0
3. Decode 4-bpp packed PICO-8 framebuffers carefully; use exact palette matching and asymmetric fixtures to catch nibble/threshold bugs
loop · negative result · ⚠ session-derived, unverified
Among Them / Coworld / Persephone pixel framebuffers cross the wire as a 4-bits-per-pixel packed byte array (8192 bytes = 128128/2) of PICO-8 palette indices, with no structured state API -- all information (phase, roles, positions, chat, offers) is extracted visually. For pixel (x,y) the byte index is (y128+x)//2 and the value is the LOW nibble (bits 0-3) when x is even/left, the HIGH nibble (bits 4-7) when odd/right, stored left-to-right top-to-bottom. Wrong nibble order silently mirrors/garbles all downstream vision, so decode one saved frame to PNG and eyeball it against the known 16-color palette before building any parser. Tests MUST use asymmetric, distinguishable fixture values (distinct high/low nibbles, ascending sequences), never uniform fills -- a decoder unpacked the high nibble first when spec was low-first but a 0x0F fill made order irrelevant so the test passed and the bug shipped. For 16-color exact-palette-index pixel-art matching with a shadow-equivalence LUT, direct EXACT pixel matching beats every standard technique because data is exactly quantized not noisy (FFT cross-correlation assumes L2 not categorical equality; integral images need separability; downsampling can't average palette indices; SIFT/ORB are poor on sparse-gradient pixel art; full-frame hashing is useless as every frame is unique). Template/shape matching can pass on busy backgrounds yet silently fail on uniform ones: sprite outlines drawn as color 0 cleared a 70% threshold on a colorful overworld but were indistinguishable from a black HUD background, so add a context flag (outline_is_black=True) accepting the bg color as valid outline and verify the matched-pixel fraction in EACH distinct background context. sources: bitworld/among_them/GRAPHICS_REPORT.md, players_checkouts/players/users/james/personal_cogs/persephone/GAME_AP, players_checkouts/players/users/james/personal_cogs/persephone/README., players_checkouts/players/users/james/personal_cogs/persephone/RULEBOO (+5)
4. Enumerate and classify every game phase; interstitials are phases, not skips, and are the one-time channel for hidden setup info
loop · ⚠ session-derived, unverified
Interstitial/intro screens are distinct game PHASES a pixel-perception player must classify, not skip: a player that only knows 'lobby' and 'playing' will misread enumerated pre-play phases (roster panel, role panels, schedule panel), and a missed special screen (voting/meeting) passes through as gameplay, manifesting as silent paralysis (no-ops for the whole phase), not an error. Enumerate EVERY phase the server can render and decide per phase what the policy does (usually: detect, extract one-time info, then idle), and validate any pixel-based classifier gate (e.g. a single >=30%-black-pixels threshold) against every special screen, not just the obvious ones. Reveal/intro screens are the one-time channel for hidden setup info (room assignment, color, sprite/shape, alliance/team, own role, full role list, schedule) shown exactly once and never re-shown -- a policy needing it must PARSE and PERSIST it during the reveal phase. Actively MANAGE the intro sequence: hold a minimum dwell per panel so perception can parse, emit idle/no-op to avoid accidentally advancing, and gate advancement on a concrete belief-state condition (all room assignments populated, my_role set) rather than a blind timer. Beware optimizing by short-circuiting gameplay sprite/actor scanners on OCR-classified interstitial frames: information visible ONLY during the interstitial (teammate identities on a role-reveal screen) is on the pixels but then never extracted, so before assuming the player knows something shown at game start, verify the pipeline actually scans that frame. sources: opencode:ses_1fb2002bcffeK2Ldmjf34wdpM3, opencode:ses_20101b42dffeJU331wyDQFEn9W, opencode:ses_2010d071bffeCAzYxkexP7hIQ8, opencode:ses_201713d2dffe7pZ6AcZMDZXvNZ (+1)
5. Don't latch role/identity on one sprite frame; require K frames, cross-check, gate the WRITE, and provide a logical fallback
loop · negative result · ⚠ session-derived, unverified
For role/identity/state classification driven by a single sprite or pixel match in a screen-reading game policy, never let one positive frame flip a latched belief field -- one sprite collision at a fixed HUD position can misclassify state. Require K consecutive positive frames before promoting, use a one-way latch for irreversible transitions, and cross-check cheap independent channels (e.g. if an interstitial OCR already asserted a value, require a stricter threshold before a HUD-sprite override). A stability gate is mathematically unreachable if the triggering screen lasts fewer than K frames, so verify the trigger window exceeds the requirement -- and first compare to existing working baselines; a simpler proven path (scan the reveal screen and stamp immediately, no histogram) may be more correct. Gate the WRITE to the belief field, not just the detector: a field unconditionally overwritten every frame has no stable value, so guard the promotion (e.g. only promote from Unknown) so a single false positive cannot flip it permanently. State derived from one brittle match also needs a logical fallback inferred from orthogonal signals (e.g. infer a no-longer-acting state from 'capability never re-armed since a phase transition'). Symmetrically, an N-frame confirmation threshold permanently LOSES high-stakes cues visible for fewer than N frames, so lower the threshold for high-stakes cues and re-arm/persist-and-recheck on the first normal frame after an interstitial ends. sources: players_checkouts/players/users/james/personal_cogs/among_them/guided_, claude-code:73fa0a3d-37fb-4408-bae8-245db6e62c56, claude-code:91f456a7-41e8-4a89-98e9-44e379160b15, codex:019e08db-c4f1-7ec1-8f76-108fbbe21008 (+4)
6. Throttle expensive fallback parses with a tick-interval cooldown
loop · ⚠ session-derived, unverified
Add a tick-interval COOLDOWN (e.g. probe every ~12 ticks / ~0.5s) when running an expensive fallback parse triggered by failed localization. Probing every tick during any perception drop (kill animation, spawn) starves the localizer's re-acquisition, while a periodic probe still catches the target screen within ~1s of it appearing. Relatedly, make perception tunables discoverable and overridable: define a match threshold as a named module constant in one obvious place AND expose it as a function kwarg defaulting to that constant, so callers that know they have a clean/unoccluded sprite can pass a stricter value. sources: opencode:ses_20101b42dffeJU331wyDQFEn9W, opencode:ses_20463f5e6ffeWi1ooj1agqH80a
7. Suspect perception before strategy and trace pixels-upward in stages; silent perception no-ops are worse than crashes -- trace consumers and alarm when a gate never fires
loop · negative result · ⚠ session-derived, unverified
When a vision-based player behaves mysteriously, an LLM reports it 'can't see' state, or a whole game state is ignored, suspect the perception/detection pipeline BEFORE strategy, and trace every layer frame -> OCR/decode -> keyword/region match -> belief update -> decision snapshot, instrumenting each stage independently: (1) OCR/decode never extracted the value, (2) extraction happened but belief/event code dropped it, (3) belief had it but the LLM/decision snapshot omitted it. A dedicated perception.jsonl recording raw OCR separately from belief localizes the break (outbound chat_sent worked but inbound chat_observed was empty -> downstream of OCR). 'Perception recognized the phase' and 'the policy acted on it' are SEPARATE failures: cross-reference the engine/server log against the policy's own trace (a server log 'vote called: lime called body' proved a vote screen existed while the policy trace showed voting:null, pinning the bug to parsing). 'X is detected but behavior never changes' almost always means the detection RESULT is computed but never wired into state -- grep for CONSUMERS of a signal, not just producers (an interstitial classifier returning InterstitialRoleReveal is useless if nothing maps it to RoleImposter; a radar-dot scanner can fill a radar_dots field task filtering never reads). A perception subsystem that silently fails to fire is more dangerous than one that crashes: meeting/vote parsing returning valid=false for every bot meant the policy never entered meeting mode, never voted, and lost to a server default, invisibly; a failed parseVotingScreen left the bot reading interstitial garbage and the voting state machine, gated on a voting.active flag the parser never set, silently never advanced. So when a perception gate guards a whole behavior mode, add an alarm/log that fires when the gate never passes during a window it should, gate state machines on POSITIVELY-CONFIRMED perception, emit a distinct diagnostic when a parser fails, and make the idle/unknown state do something observable (gentle exploratory movement) so 'warming up' is distinguishable from 'stuck'. Concrete chained failure modes seen: messages stacked from the bottom (y=104,111) while the parser scanned top-down (y=10,17); a leading sender sprite shifting text start x=2->x=13 past a space-skip peek; matching the engine log string ('shared roles') instead of the rendered string ('ROLE XCHG:'); a silently-dropped None lookup (decode_player_index returned None so pending_entry stayed None and entry was never granted). When OCR reads a value correctly but the parsed field is None, the bug is DOWNSTREAM in the parse/offset step. Before debugging any detection bug, confirm the expected signal even exists in the stream (dump a raw frame at the moment the server log says X happened and confirm the pixels are present AND parsed), add graceful fallbacks (color-only resolution), and guard every sprite/region lookup that can return None before slicing the frame. Often the real fix is exposing the right structured fields (alive_players, dead_players, self_can_vote), not prompt wording. sources: claude-code:3b14ac5a-1d66-4524-94c0-59a48466d223, claude-code:73fa0a3d-37fb-4408-bae8-245db6e62c56, claude-code:91f456a7-41e8-4a89-98e9-44e379160b15, claude-code:aae25940-ade3-41bc-b817-517dc86ccebf (+15)
8. Triage perception regressions with git and replay artifacts, not edits
loop · ⚠ session-derived, unverified
When perception 'worked recently' but now fails, separate 'the game changed' from 'our code regressed' BEFORE touching code: run git log -G on the parser and confirm the file is not dirty. A green local suite can be false confidence when the test uses a stale fixture (a color-ordered fixture passed while the real change was switching to the public coworld package/image flow). Reconstruct the exact failing frame from the replay artifact and run the real parser gates against it from a DISPOSABLE probe OUTSIDE the repo (a throwaway Nim probe under /private/tmp that replays the killed episode to the failing tick) -- replaying replay.json to the specific vote tick yielded a precise 'first_mismatch slot=0 expected=0 got=2' pinpointing the failing gate in one shot, while keeping the working diff clean. Keep replay artifacts as ground truth for perception debugging. sources: codex:019e2886-d576-7770-be1b-6d92ca97761e
9. Diagnose a perception bot via the event trace first, raw frames second
loop · ⚠ session-derived, unverified
When a perception-driven game bot's behavior drifts after a fork or update, suspect game-graphics changes (sprite atlas, font, room coordinates, UI/panel layout) before bot logic, and diagnose via the EVENT TRACE first, raw frames second. The signature of stale perception is correct PRECONDITION events still firing (e.g. an action-trigger event) while the downstream screen-TRANSITION event is never detected (the transition/interstitial event never fires) -- missing high-level events localize the break to perception/state-detection; conversely spurious high-frequency events (a cooldown firing hundreds of times, excessive state-change events) indicate a stuck or oscillating state. Only after the trace narrows it, dump the raw framebuffer around the suspect tick and render it as an image so a human can visually confirm the HUD/layout, mapping event time to frame index via fps -- but remember the bot's internal frame tick can diverge from the capture index when frames are dropped. sources: opencode:ses_20b1744aeffettq8pxHNVgjKNC, opencode:ses_20b179f2fffeJe4vpbx2D4tWuR, opencode:ses_21f67d3f5ffeBF92agcd0cbOV0
10. Build a frame recorder first; test against real captured frames not synthetic/noise, and validate at fixture AND live-match levels
loop · ⚠ session-derived, unverified
Build a frame/observation recorder BEFORE debugging perception so you capture exactly what the agent saw, not what you assume: dump every tick's raw frame to disk (e.g. concatenated 16384-byte uint8 frames in frames.bin plus per-frame decisions.jsonl, events.jsonl, reflexes.jsonl, manifest.json). Build perception tests on REAL sim-rendered frames (a capture client recording a live episode's frame stream plus a timeline viewer), because synthetic frames don't reproduce calibration/palette drift, and build the harness with known-good expected outputs BEFORE fixing bugs (e.g. ~42 tests over ~10 live fixtures) or you cannot tell whether a fix regressed another view. Feeding RANDOM-NOISE frames is a weak signal: random pixels neither localize to the real map nor reliably trip the same code paths as real frames, so 'it returns noop on noise' proves almost nothing. Validate a perception change at TWO levels: fixture/unit tests on real captured frames prove the parser matches/rejects, and a full live match proves detection actually triggers the mode transition end-to-end -- passing fixtures alone does NOT prove behavior changed. Also live-verify against a real server with a fixed seed: connect a WebSocket client, call parse_frame() per frame, and print the view PLUS every extracted FIELD (run both a 1-player lobby and a full 6-player game to trigger transitions), because per-field printing exposes partial failures a view-label-only check misses (a 'role=None, team=Shades, room=None' read surfaced a centered-text extraction gap). Make diagnostics ASSERT, not print: require real positive fixtures to parse valid AND representative negative fixtures (gameplay, interstitial) to be rejected, so the threshold cannot drift to always-true/false. Commit each fixture as a frame (.npy/.bin) paired with a .json of expected assertions; strengthen integration tests beyond single-frame parsing with temporal stability (same view across N repeated frames), state-transition cleanup (alternating playing/whisper frames clearing stale chatroom state), and an exhaustive synthetic matrix (every shape x color x {normal, shadow, partial-shadow}); use explicit skip markers tied to fixture availability so a test announces a coverage gap and auto-activates once the .npy fixture is captured. Land small isolated offset/flag fixes first (one constant each with a focused unit test asserting corrected coordinates) before a larger structural change so a regression bisects to one edit, and prefer a standalone per-region sub-parser module over burying region logic in a general parser. Before writing assertions, probe canned fixtures through the real parse_frame to learn which fields actually populate ('raw frame has X but belief lacks X' is expected layering, not a bug). Caveat: fixtures captured DURING a bug encode the OLD buggy expected outputs, so after a fix update assertions to the corrected truth (RoleCrewmate -> RoleImposter) rather than reverting, and a downstream mode transition (ModeTaskCompleting -> ModeHunting) confirms the fix flipped behavior. Your opponent/filler-bot policy shapes which sub-states you can observe, so some fixtures must be choreographed (a waiting_entry view couldn't be captured because aggressive fillers always initiated whispers first; fix with less-aggressive fillers or two-bot choreography). Pin concrete numeric edge cases for capture tooling: change-detection threshold boundaries (at vs just-under), all-black vs all-white max-diff frames, identical consecutive frames (zero diff), and the first frame (no prior to diff). sources: claude-code:fa645b7b-4fa0-4b55-992e-b273ee703391, codex:019dffc3-a08f-76b1-8b07-8a16f6e5ae38, codex:019e004f-1647-77a1-9e58-fa29c8f8ef89, codex:019e0691-0971-7623-9fb2-5b1aed778856 (+10)
11. Diagnose frame delivery/latency before blaming a parser; prefer the server-authoritative signal over local estimators
loop · negative result · ⚠ session-derived, unverified
Frame delivery lags the game-state tick and can stall, so diagnose delivery before blaming a parser or threshold. HUD/visual signals lag server state by one+ frames (a kill cooldown set on tick N is visible as the shadowed sprite only at N+1+), so don't treat a freshly-changed pixel as instantaneous truth -- allow slack or confirm against an independent signal. A server may resend an identical frame for hundreds of ticks (observed 360+ voting ticks holding ~2 unique frames), so count UNIQUE frames over the window before concluding a parser failed; detect frame-TYPE switches by diffing a cheap per-frame statistic (black-pixel count jumping ~7%->~93% in one tick signals a server-side switch, not a gradual transition). Prefer the server-authoritative rendered signal over a shadowing local estimator and delete the duplicate: a killCooldownRemaining field never updated (always 0) shipped to the LLM as a constant lie while lit-button perception was the real kill_ready gate -- when a binary perception flag never flips, check the server's draw code for the exact ready condition before adding state. Before adding new protocol, check whether the renderer's info is already decoded and thrown away: a vote-timer was collapsed to timer_present:bool, discarding SpriteDef width the decoder already keeps, even though a shrinking-width bar is a drift/latency/early-close-proof server-authoritative fraction-remaining strictly better than a local frame counter. sources: claude-code:067e7439-cbd1-4b26-9b3f-d054bde3aa40, claude-code:3837ef5e-3e9c-41bc-b290-df5865581698, claude-code:4ec865f5-bcbc-44f1-b109-a27d69f55506, claude-code:73fa0a3d-37fb-4408-bae8-245db6e62c56
12. Treat the renderer source as the spec for every pixel offset; invert algebraically and accept multiple known scan positions
loop · negative result · ⚠ session-derived, unverified
Treat the game's renderer source (border colors, title position, column offsets, per-row stride, label format, alignment anchors) as the authoritative spec for every pixel offset a parser reads -- not parser constants, docstrings, or screenshots. Derive the inverse algebraically: if the renderer draws a feature at (sprite_x - dx, sprite_y - dy), the parser must recover sprite = (feature_x + dx, feature_y + dy); an off-by-one (+3 vs +4) silently misreads state with no crash. In Crewrift Sprite-v1 specifically the DRAWN object position is NOT the collision/interaction point (entities drawn at (x - SpriteDrawOffX - 1, y - SpriteDrawOffY - 1) while report/kill/collision logic uses raw x/y, a (+3,+9) offset), so add the collision offset uniformly in the resolve step or every distance check is silently off. Right-aligned/width-dependent HUD text means start-x depends on rendered width, so compute scan bounds from the alignment anchor minus max text width (start = anchor - max_chars*char_width) and scan multiple x-offsets for centered titles. Fixed-coordinate parsers silently break when a client UI moves between versions (a 'WHISP' header moved (2,2)->(42,2) so the agent read the round clock and misclassified the view as PLAYING): ADD the new scan position while keeping the old one (accepting multiple known positions kept 475 tests green and tolerated drift). You can encode and test a phase against the renderer with synthetic frames reproducing the exact layout instead of waiting for a live session to hit a rare phase. The same 'port the server's exact algorithm' rule applies to off-screen target projection: a naive clamp-to-screen-rect is wrong for diagonally off-screen targets, so port the server's reference ray-clip from the player's screen position toward the target (handle the on-screen case separately). A clipped radar/minimap pip encodes only DIRECTION not distance, so reconstruct a world-space ray with origin at the agent's world position and direction = (pip_screen - agent_screen) (camera translation is pure), and add a t>0 forward check to the ray-AABB intersection or the ray 'hits' regions behind the agent. sources: personal_labs/crewrift_lab/docs/crewrift-player.md, claude-code:3b14ac5a-1d66-4524-94c0-59a48466d223, claude-code:4430a077-5411-426b-a87a-75512ec4300f, codex:019dfef8-dc40-7273-b85a-04aa3a51ff07 (+4)
13. Derive on-screen positions from ONE shared server-matching formula and an exact noise budget, not margin-expanded boxes
loop · negative result · ⚠ session-derived, unverified
Use ONE canonical world-to-screen anchor formula per object class in a screen-reading game policy and match the scanner's detected icon in that SAME coordinate space; audit every conversion path so they agree, because when one code path gates visibility and another does the actual match, the gate disagrees with the matcher and causes premature/false triggers (e.g. an 'expected screen position' helper used a target's geometric center while the icon renders a fixed ~14px above the rect). Match by the server's exact render offset, NOT a margin-expanded bounding box: expanding each target rect by a fixed margin overlaps rects in dense layouts where targets sit close together, so one icon matches multiple targets and selection picks wrong; since the icon sits at a fixed offset, the rect is recoverable EXACTLY. Derive tolerances from the actual noise budget, not round numbers (icon-center offset + per-frame bob + screen quantization + position quantization), and document each term. A single-exact-pixel probe silently disables a whole feature if off by a few px (a self-color probe returned -1 for thousands of frames probing the wrong anchor), so derive the anchor from the rendering source and probe a small tolerance window, validating with a two-line manual arithmetic check on the exact failing case. A policy thrashing on a target it can never satisfy is usually ambiguous belief object-to-target attribution, not the action/nav policy. sources: codex:019e0105-2b01-71a3-8986-eb1e5269ea83, codex:019e04d1-5052-7b42-907f-c0a2e3d539a8, codex:019e05eb-dbd6-76f0-8bf4-3ef821cb7cc6, codex:019e132d-8773-7503-86cd-a77b0a8d21eb (+5)
14. OCR fails live after a font migration: source glyph metrics from the sim and reuse its font engine
loop
When OCR works in tests but fails LIVE, suspect a font/rendering mismatch, not the parsing logic. The classic breakage: a stale hardcoded glyph stride (a fixed 7-pixel stride while the engine migrated to a variable-width font with dynamic advances and a font-file-derived background color), silently garbling reads; modulabot's chat OCR mis-read every line until ported to the shared variable-width tiny5 engine (and its panel scan was off by one row, chatY+2 vs chatY+1, dropping first lines). The font path is often a compile-time constant while the font file loads at runtime, so check git log for font/OCR commits and compare binary build timestamps against them. Fix by REUSING the renderer's own font decode helpers and sourcing constants (text X offset, glyph stride, char count, panel width/start row) FROM the simulator, not per-bot copies. Beware glyphs composited by the NATIVE CLIENT (a chat input box from a separate ascii.png sheet drawn atop the framebuffer): they never appear in the server-sent frame, so a wire-frame parser must not expect them. sources: bitworld/among_them/GRAPHICS_REPORT.md, bitworld/among_them/players/modulabot/TODO.md, opencode:ses_21b008642ffeCM6Q48G44aM6k0, opencode:ses_21eeac2baffegQWJK4QL6GuTEI (+2)
15. Keep a documented legacy coordinate fallback so renderer moves do not break the parser
loop
Pixel/HUD parsers tied to renderer-specific screen coordinates break whenever the game renderer moves elements; keep a primary coordinate plus a documented legacy fallback so both old captured fixtures and new live frames parse (e.g. a chatroom-header sprite at x=66 for the current renderer, falling back to x=22 for legacy fixtures). sources: players_checkouts/players/users/james/personal_cogs/persephone/TODO.md
16. Gate visual detectors on positive confirmation of the right screen; rank sources by confidence with hard-over-soft and a behavioral fallback
loop · negative result · ⚠ session-derived, unverified
Gate a heuristic perception detector on POSITIVE confirmation of the right screen context (e.g. OCR of a title/banner) before letting it run or write belief state, because presence-only heuristics false-positive on screens that legitimately show all entities (a color-pixel scan that ran on every interstitial - including a lobby that shows all sprites - wrongly concluded all bots were imposters). Localize premature-firing bugs by dumping per-frame phase classification and comparing when state CHANGED vs when the triggering frame actually appears. Establish a confidence hierarchy among perception sources: a HARD positive signal (OCR-read literal text) overrides a SOFT one (a single-position sprite/HUD match), and never let a soft exclusion block a confirmation the agent can directly observe; gate soft signals behind 'is the hard signal currently unavailable?'. Build role/state detection in redundant layers keyed to different phases (an early OCR-banner primary plus a gameplay-time HUD-icon backup, defaulting to the common case when neither fires), and when pixel-perception of identity is fragile add a BEHAVIORAL fallback driven by action outcomes (after repeated failed actions against the same target, infer it is friendly and stop targeting it). sources: opencode:ses_1f6f76557ffeCfmMTG7BPJwGRE, opencode:ses_1f7b6dc30ffeDmQtlUZI25HAjh, opencode:ses_1fc4cec1bffe16bZw4eHP3SDBL, opencode:ses_1ffe75d89ffeHvW4p2ARluHh51 (+2)
17. Build an explicit observability catalog phase-by-phase and make last-seen memory mandatory under partial observability
loop · negative result · ⚠ session-derived, unverified
Before writing a policy for a partially-observable game, build an explicit observability catalog classifying every state element per phase as always-visible, conditionally-observable (LOS/viewport/one-time-reveal gated), or never-directly-visible (with a 'when revealed' note); drive perception+memory architecture from a gap table of {view? detection? parser? key gaps}, because a missing-phase extraction silently destroys foundational belief state (a roster-reveal phase with no parser lost all player->room mappings). Persist anything conditionally visible into last-known-sighting memory the instant it is observed - memory is mandatory, not an optimization, because re-reading the live view each tick loses an entity once it leaves sight. Deduplicate raw per-tick sightings keyed on BOTH time and space, with constants chosen deliberately since they control how trigger-happy downstream inference is. Distinguish 'the game does not expose X' from 'our code does not parse X yet' before discarding a feature, and treat the latter as an upstream perception dependency to implement first (documented so behavior pauses to collect that data) rather than permanently assuming X unavailable. Ground any claim that an entity/objective belongs to the agent in a server-rendered perceptible cue, not inferred behavior. sources: claude-code:067e7439-cbd1-4b26-9b3f-d054bde3aa40, claude-code:0e7b14ca-ffd5-4137-9456-47485d3c6f87, claude-code:d4cde1e2-f4b5-4bd6-bf05-dd48002edf58, opencode:ses_1fb1787afffemhwtF09UMzPc5V (+3)
18. Fuse multi-fidelity perception channels with per-cell provenance; never overwrite confirmed data with low-confidence hints
loop · negative result · ⚠ session-derived, unverified
Perception channels deliver the same entity at different fidelity (a precise nearby sighting vs a low-res far minimap dot). Feed all position-bearing channels into one unified known-positions view downstream modes consume (reading only the high-fidelity channel makes the agent blind outside immediate range), but do NOT pollute the authoritative entity registry with low-confidence observations: keep speculative sightings (minimap dots) in a separate clearly-labeled last-seen lane. When building an occupancy/belief grid from sources of differing reliability, track provenance per cell (e.g. a viewport_confirmed boolean): high-confidence direct observation may overwrite anything, low-confidence hints may only fill cells still UNKNOWN, and shield immutable cell types (WALL, hub, stations, extractors) from floor overwrites. Write a cell only on positive evidence (its position appears in grid_tags/grid_inventory/grid_agent_ids); absence of tokens means 'unobserved', NOT 'empty floor', and since the engine emits tokens only inside a ~circular ~6-cell vision radius while the buffer is a full 13x13 square, iterate only over positions actually present in the parsed dicts or the corners overwrite real data with bogus floor. Track stationarity with a movement threshold (>2px resets the counter) and per-entity tick stamps (cleaner than a separate visibility flag); treat reachability discs only as a NEGATIVE signal (where an agent cannot be), inferring heading from observed motion over dozens of ticks. Don't let code needing a fully-resolved handle silently block acting on a visible-but-unidentified target -- add a fallback that interacts on reaching its last-known position, and prefer a blind state-free action when one exists (a 'blind offer after entry request' completed exchanges that perception-gated WAITING_ENTRY pixel parsing never could). sources: codex:019e031e-f7d3-7342-a7d9-96993bc91097, codex:019e067b-643a-7100-abdb-8d6a0c79d242, codex:019e08cf-4f50-7ef1-bc11-82b8d92354b8, codex:019e08e5-547c-7da1-b1de-44d0b4133972 (+3)
19. Use absence of a cue to prune hypotheses, but tag evidence absolute vs positional, gate on prior confirmation and grace, recompute each frame, and debounce flicker
loop · negative result · ⚠ session-derived, unverified
Use the ABSENCE of an expected perception cue as a first-class signal to PRUNE belief hypotheses, not just presence to confirm: if an off-screen task's projected radar-dot position stays far from every detected dot, that task is probably not assigned. But explicitly tag each negative-evidence source as ABSOLUTE vs POSITIONAL/directional and never write a positional signal into a permanent/terminal belief state: absolute ('I stood there and saw no icon') is permanent; directional ('from here no radar pip points at it') is transient and reversible because its value depends on where the agent is standing -- conflating them latched directional pip-exclusions into the same one-way terminal state as absolute icon-miss evidence, so tasks excluded from one position stayed dead forever even after moving, leaving the bot idle. An affordance DISAPPEARING can mean SUCCESS, not absence: the 'missing icon = wrong task, prune fast' rule wrongly pruned a task whose icon vanished because the action SUCCEEDED (the server accepted the task while the bot recorded zero task_completed events), so gate negative-evidence pruning on prior positive confirmation and shield positively-confirmed targets (state==confirmed) from fast miss-pruning while leaving speculative targets unshielded. Guard the inference: run it only when >=1 indicator is detected this frame (with zero signal you cannot distinguish 'far from all dots' from 'no perception'); gate on N consecutive confirming frames with a separate tunable distance threshold (one 'no-match' frame is unreliable; 2 frames was too low, ~6 frames/~0.25s absorbed noise); reset the counter both when localization is lost and the moment contradicting positive evidence reappears. Across a channel handoff (far radar pip -> high-detail icon) expect a frame gap: the pip vanished the instant the player got close but the full icon took 2-3 frames to scroll in, so add a grace period (PipDisappearGraceTicks=5) keyed to pip count dropping to zero. Debounce perception against single-frame flicker rather than reacting to it and recognize PARTIAL targets: a task icon detected one frame and missing the next (bob, clipping) should not immediately resolve gone -- raise the miss-resolve threshold (TaskIconMissResolveFrames 2->6), and if ~70% of an icon shows, recognize it (full-icon-required creates a window where the target is dropped between radar-pip loss and full-icon registration). For a flag meant to reflect 'true right now', recompute it from scratch each frame rather than accumulating in a saturating counter (a counter that saturated after ~12 frames latched and stopped tracking reality), and clear any transient per-entity flag at the TOP of the per-entity loop before any 'continue'. 'For each task, is there a pip near its projected dot?' and 'for each pip, which tasks lie along its ray?' are geometrically equivalent under exact projection, but the per-pip framing naturally yields a TRANSIENT exclusion set rather than a per-task accumulating counter -- the framing is a real design lever. Prefer fixing the upstream setter of a sticky/permanent flag over making it forgiving/expiring (resolvedNotMine was made accurate by fixing its setter, not adding expiry; a sticky flag is acceptable only if set accurately). When negative evidence IS reliable and you know the exact deterministic offset where evidence should appear, check that point with a couple-pixel tolerance and treat absence as INSTANT resolution (~24-consecutive-empty-frame threshold dropped to ~2). Tightening a resolution threshold yields nothing if a blanket shield still exempts entities (dropping missing-frames 24->2 did nothing); audit every caller that exempts entities from a decay, and beware the race where the policy holds N ticks but the server confirms at K<N (held 84, accepted at 72, then 24 confirmation frames ran with the signal gone) -- trim hold time to just past the acceptance threshold. sources: codex:019e0140-e554-78e3-b66f-80634cc63f5d, codex:019e03bc-5520-7e72-aa5c-a572ead62b91, codex:019e03bd-5f9b-7482-a03a-24cb99c6c7ed, codex:019e0436-7b17-7433-b3d9-5c78fad74312 (+8)
20. Key edge-triggered reflexes on stable identity, not visible-entity counts; require true rising edges
loop · negative result · ⚠ session-derived, unverified
Edge-triggered reflexes keyed on a raw count of currently-visible entities have no identity memory and re-fire on the same stimulus when an entity leaves and re-enters the viewport (a bot fleeing its own kill victim re-triggered each time the body left and re-entered view). Key reflexes on stable identity (remembered positions within a radius) and clear the memory set at round/phase boundaries. Perception gaps also create false hard evidence: a 'newly detected at X' rule fires on scanner dropouts, so require 'newly appeared from elsewhere', gate on the previous frame, and suppress reappearances of a recently-seen nearby entity. With rising-edge (0->1) input detection, hold the press long enough for the server's edge (2+ frames) and gate the action by valid view so it stops emitting once the target view is reached; when one control is overloaded by context, observe the relevant world state and pick the correct button once rather than blindly cycling. sources: claude-code:067e7439-cbd1-4b26-9b3f-d054bde3aa40, claude-code:3b14ac5a-1d66-4524-94c0-59a48466d223, codex:019e1591-004e-7040-b768-062bd8e6acd4, codex:019e1593-c00d-7a32-9dfa-1cf6b16b01a4
21. Port a game-policy perception layer behind a byte-for-byte parity rig against the original as oracle
loop · ⚠ session-derived, unverified · see related: U0730 (other tier)
When porting a game-policy perception layer to a new language/runtime, build a byte-for-byte parity rig that uses the original implementation as the ground-truth oracle: nothing merges into the new perception tree until the parity suite is green on a fixed fixture set. Decide the oracle mechanism up front (a small CLI reusing the original vs instrumenting the original), capture 50-100 fixtures spanning every game phase (lobby, role-reveal, playing, body-sighted, meeting, voting, results, interstitial), and port the smallest dependency-free module first in strict dependency order, gating each module with parity checks (e.g. ~160 checks across 10 fixtures). When the runtime forbids the original toolchain, parse the raw packed pixel frame directly rather than the structured state vector (lossy for many perceptual facts), allowing a documented state-vector tap only for a field lossless only there. Do NOT re-derive behavior from prose docs; assert equality against captured reference output. Commit each fixture as a locked triple (raw frame, capture metadata, reference truth) stamped with the asset/atlas hash so the harness rejects fixtures computed against a different asset version. Consume the original's shared vectorized kernels so behavior stays bit-aligned, verify EVERY signal the reference consumes survives your simplification (dropped 'redundant' bookkeeping can be load-bearing), and diff against a sibling port that already migrated. Ground 'what is rendered' in the actual load/render path, not the asset directory. Optimize correctness first and defer JIT/numba/vectorization until a measured budget forces it (cold-start can blow a tight player-init deadline). sources: players_checkouts/players/archive/README.md, players_checkouts/players/archive/players/among_them/coborg/DESIGN.md, players_checkouts/players/archive/players/among_them/coborg/PLAN.md, players_checkouts/players/archive/players/among_them/coborg/README.md (+10)
22. Port a perception layer behind a byte-for-byte parity rig against the original as ground-truth oracle, module-by-module in dependency order
loop · ⚠ session-derived, unverified · see related: S021 (other tier)
Port a perception layer to a new language behind a byte-for-byte parity rig that uses the original implementation as the ground-truth oracle: nothing merges into the new perception tree until the parity suite is green on a fixed fixture set. When a player must run completely in Python (no Nim toolchain in the runtime image), parse the raw packed pixel frame directly (pixel-first) rather than relying on the structured state vector, which is lossy for many perceptual facts -- the one allowed exception is a documented 'state-vector tap' for a field lossless only there (e.g. exact task-progress percentage). Decide the oracle mechanism up front (a small CLI reusing the original perception vs instrumenting the original bot), capture 50-100 fixtures spanning every game phase (lobby, role-reveal, playing, body-sighted, meeting, voting, results, interstitial), and port the hottest smallest dependency-free module first, in strict dependency order (interstitial and ignore are independent and cheap; localize gates ocr, voting, and task-icon scanning -- a working order is interstitial -> ignore -> localize -> task-icon -> ocr -> voting), rolling the sidecar schema version forward as each field lands and gating each module with parity tests against the Nim ground-truth oracle (e.g. 160 parity checks across 10 fixtures). Do NOT re-derive behavior from prose docs (lossy and slow); assert equality against captured reference output (reshape/frombuffer the source .bin and assert equality can substitute for visual inspection of a ported asset). Commit each fixture as a locked triple in one commit -- raw input frame (.npy or palette-indexed .bin), capture metadata (.meta.json), reference ground truth (.nim.json computed by the original) -- and stamp each with the sprite-atlas/asset hash so the harness rejects fixtures whose truth was computed against a different asset version. Treat the original as a PARITY REFERENCE and SPEC: consume its shared vectorized kernels (sprite matching, task-icon scan, glyph/text match, localization) so behavior stays bit-aligned, reimplementing a scan only when none exists and it is trivial (radar dots = yellow pixels in the border ring); pin the port against the original's exact outputs on shared fixtures (same camera lock, e.g. (504,54)) before adding behavior; verify EVERY signal the reference consumes survives your simplification (a port dropped the reference's radar-dot/checkout bookkeeping assuming it was redundant, but the pixel adapter only surfaced the mandatory icon signal, so the dropped logic was load-bearing); and diff against a SIBLING bot that already did the migration (italkalot/ivotewell for modulabot). When a parity test fails on an unrepresented phase, capture a fresh fixture and confirm the bug reproduces BEFORE fixing. Use the canonical UPSTREAM repo for assets (map JSON, spritesheet, palette, aseprite), not a derived copy that silently drifts, and ground 'what is rendered' in the LOAD PATH (what the sim init loads and render() blits), not the asset directory -- several sprite/aseprite files were never loaded, so reading them as live would mislead. Optimize correctness first and defer JIT/numba/vectorization until a measured budget forces it (numba cold-start can blow a tight ~5s player-init deadline). The archived Among Them 'Coborg' player preserves the full v5 pipeline (frame, sprite_match, actors, geometry, ignore, interstitial, localize, ocr, tasks) plus the parity harness and a Coworld policy bridge -- use it as the reference when porting or comparing. sources: players_checkouts/players/archive/README.md, players_checkouts/players/archive/players/among_them/coborg/DESIGN.md, players_checkouts/players/archive/players/among_them/coborg/PLAN.md, players_checkouts/players/archive/players/among_them/coborg/README.md (+10)
23. Verify perception field schemas against live code, never the brief; re-derive game-flow phases/timers/limits from source
loop · negative result · ⚠ session-derived, unverified · see related: S023 (other tier)
Before writing code that consumes an external schema, re-derive every documented game-flow phase, timer, and limit from the actual game source rather than the brief - one audit of a game found missing phases (RosterReveal, LeaderSummit), a '5-second countdown' that was 1s (24 ticks), a role-reveal '5 seconds' that was a 15s 4-panel intro, and a '36-char' chat limit that was actually 58 chars. sources: codex:019e05e2-eadc-70a1-a58f-add2c62f1046, codex:019e08bb-a4ea-7c83-a8d6-2652eaa1a3b8, opencode:ses_1ffc701efffeS5c5ueXd2vWIkD
24. Keep perception schemas additive, treat fallbacks as code smells to revisit, and log a single converged JSONL trace
loop · ⚠ session-derived, unverified
Extend a perception dataclass ADDITIVELY when upstream exposes more than it carries (keep existing fields, add new ones like sprite shape/position alongside color) so downstream policy reading old fields does not break -- backward-compatible schemas decouple parser upgrades from policy rewrites. When a shared sprite reader has blind spots on specific shapes (read_sprite() misses centerless shapes like a ring), layer a fallback (try the shared reader, then explicit shape detection plus a dominant-fill-color read in the bounding box) rather than modifying shared sprite behavior and risking regressions. But treat any perception fallback as a code smell to revisit, not a permanent fix: resolving a player 'by color' was flagged 'almost always not a great sign,' and once the underlying outline-color-0 shape detection was fixed the color fallback became dead code -- file the root cause, fix the real detector, verify the fallback is now redundant. Log the FULL per-frame perception output to one JSONL trace (camera localization+score, self position/color/role/ghost, detected actors/bodies/ghosts, task icons, radar dots, ignore-mask pixel count, interstitial flag, voting parse) emitted at the SINGLE convergence point where all phases merge, so downstream tooling reconstructs exactly what the policy saw without re-running perception and the record reflects fully-resolved (not mid-pipeline) state. Scope perception work to perception assets only: grep which modules actually import an asset before assigning it to a phase (nav_paths.bin/nav_graph.json were imported only by navigation/planner modules; defer an OCR-only font blob to the phase that first needs it). Capture a 'visual bootstrap' of live frames during a generated player's first run (dump per-frame packed .bin under visual_bootstrap/live_run/frames/) to give the downstream decoder-generation agent ground-truth data. sources: auggie:5d1ff8d5-2505-4dca-af45-56d3070363e2, claude-code:3b14ac5a-1d66-4524-94c0-59a48466d223, claude-code:7b66497b-650c-4bba-b410-6c854f59163d, codex:019dfef8-dc40-7273-b85a-04aa3a51ff07 (+2)
25. Reset belief fully (including ad-hoc attrs), reject malformed readings, and accumulate sticky channels rather than overwrite
loop · ⚠ session-derived, unverified · see related: S024 (other tier)
When perception feeds geometry into downstream logic, treat invalid perceived inputs as invalid rather than real data: a role-reveal frame reporting a non-positive room_size was accepted as a real zero-size room, so guard against malformed readings before they produce a degenerate grid. Model runtime-variable game settings as belief, not constants (whether task arrows show is a per-map setting - detect it by observation, use arrows when present, fall back to a full sweep when absent). Do not mark an objective complete merely because it left the visible set; identify the authoritative completion signal (a task bubble disappearing while inside the task rect) and beware progress fields that ramp toward but never reach 100. For sticky channels like chat where the same line stays visible across frames, key structured updates off newly-APPENDED messages not currently-visible ones, accumulate-and-dedup rather than fully replace, and match semi-structured system messages with case-insensitive substrings not brittle exact-match. sources: claude-code:4430a077-5411-426b-a87a-75512ec4300f, claude-code:48add98e-a83d-4348-97bf-0079c48c42d6, codex:019e03bf-982c-7fc0-8ba9-d4c640feb779, codex:019e04e9-c357-7c71-a255-929f3a97fda1 (+3)
26. Dead-reckon from the engine-observed last_action, not the action you emitted
loop · negative result
For dead-reckoning position in an egocentric-observation policy, use the engine-observed last_action token as the PRIMARY movement source, NOT the action you emitted. Root cause of a corruption bug: in an async live runner that observes-then-applies the latest queued action with no per-step wait, the emitted action LAGS the engine-applied one, so dead-reckoning from the emitted move pairs a movement result with the wrong action and writes immutable landmarks at shifted coordinates, creating duplicate hubs/stations/walls that corrupt pathing. EXCEPTION: when the engine reports noop for a blocked/invalid emitted move, use the EMITTED move only to mark the blocked cell, never for successful dead-reckoning. Treat type:wall as a permanent blocker but type:agent as temporary traffic, and store timestamps with observed tags because junction control mutates over time.
sources: archive/cogames_playground/alpha_cog/docs/cogs-v-clips-strategy.md, archive/cogames_playground/bulbacog/designs/INNER_LOOP_IMPL.md, archive/cogames_playground/bulbacog/designs/LOCALIZATION_ROOT_CAUSE_RE
27. Infer spatially-gated quantities from globally-available tokens, and keep your own id in the teammate set
loop · see related: S028 (other tier)
When the engine exposes global team-level inventory tokens (team: counts) for some quantities (hub element counts oxygen/carbon/germanium/silicon) but surfaces others (hub hearts) only when spatially adjacent, INFER the gated quantity between visits from the global counts plus the known crafting cost (heart craftable when >=7 each) rather than treating it as unobservable. Relatedly, keep an agent's own id in its own observed-teammate set so an agent that sees no teammates by the assignment tick falls back safely (assumes it is the highest-id visible and takes the lead role) instead of failing on an empty set. sources: archive/cogames_playground/bulbacog/designs/MEMORY.md, archive/cogames_playground/bulbacog/designs/STRATEGY_V2.md
28. Detect CvC interactive cells via a precomputed tag set; read teammate hearts from inventory
loop · ⚠ session-derived, unverified
Detect interactive cells cheaply during CvC perception by precomputing a frozenset of interactive tag IDs once (extractor, hub, junction, gear-station tags) and intersecting it with each observed cell's tag IDs; resolve the tag IDs once in the policy and pass them down rather than re-deriving the name-to-id mapping every tick. The interactive-cell set grows monotonically because interactive objects do not move. For a visible teammate, read heart count from the inventory observation (inv:* tokens parsed into a per-position inventory, stored as last_known_hearts), NOT from text. Use a talk field like h= only for commitments that outlive visibility: once an agent leaves view last_known_hearts goes stale, so the h= carried in a role claim keeps the commitment's heart context for a TTL. sources: codex:019e14de-cdca-7482-88fa-81716ee830ae, opencode:ses_1ffa96717ffeq1J1GeKsUpSp3k
29. Gate stale observations and resource claims on a TTL, but keep ownership-change detection sticky
loop
Gate ghost/stale-POI chasing and resource claims on a staleness TTL: is_stale is (current_step - last_seen_step) > ttl (strict, so the boundary is not stale), and published claims auto-expire after claim_ttl unless renewed so a stuck or dead agent cannot lock a resource forever. Separately, keep ownership/scramble detection sticky across intermediate transitions -- track that a junction was once friendly so a cogs->clips->neutral sequence still fires a scramble alert, suppress alerts whose prior observation is older than the freshness window, and leave alerts un-removed on consumption so multiple racing agents can respond. sources: optimizers/.cursor/skills/scripted-navigation/reference.md