RTK_dev_for_cm-loc

Imported from Worshisy/RTK_dev_for_cm-loc (AGENTS.md). Install upstream with npx skills add Worshisy/RTK_dev_for_cm-loc. Copyright stays with the author.

AGENTS.md — canonical state, gotchas, field-debug playbook

Review status: ⏳ Unreviewed (default; update when Yi reviews)

✍️ Claude-authored 2026-08-03. Read this first if you are an agent landing in this repo — especially on a field PC debugging a live RTK session. Everything below was measured on the actual hardware (sessions 2026-07-28 → 08-03); nothing is speculative.

What this system is

1 base + 6 rovers, ArduSimple simpleRTK2B (u-blox ZED-F9P) + Digi XBee SX/PRO-SX radios. Base broadcasts RTCM over 900 MHz; each rover computes UBX-NAV-RELPOSNED (base→rover NED vector) — that is the ground-truth product for RF-localization experiments. Campaign style: one-shot tests at unknown sites; relative location is what matters, base absolute is only a rough bonus (survey-in result ≈ meters).

Reading order for a fresh agent: this file → ROVER_ROSTER.md (which board is which) → FIELD_LED_REFERENCE.md (LED truth table) → FIELD_DASHBOARD_CHECK.md (rover web-dashboard go/no-go) → NEW_ROVER_SETUP.md (bring-up runbook) → note.md (system understanding) → RTK_setup_notes/ (deep reference).

On field Mac minis, the rover monitor is managed by ../LRLocal-Mac-EnvSetup/field-000-jobs.sh start rtk (tmux session rtk, dashboard http://<mini-ip>:8000, process log ~/field-logs/rtk.log). Session DATA is recorded by the monitor's --log-dir (the jobs script passes $RTK_DIR/data): per-session relposned_<stamp>.csv (time+loc, unfiltered — gate on carrSoln=2 offline) + .debug.log (everything). No Session recording: line at startup = display-only run, no data saved. Gotcha: the jobs script's stray-instance guard aside, a monitor started outside tmux collides on port 8000 AND the serial port (gotcha #8).

Canonical config (all verified on hardware, dates inline)

Hardware identity map (chip IDs, XBee serials, USB-bridge serials): see ROVER_ROSTER.md. USB port names are NOT identities — three different boards have enumerated as usbmodem21201 on different days (macOS names follow USB topology). Always identify a board by tools/board_id.py.

The toolbox (tools/, stdlib + pyserial only)

On the base, mon_comms.py shows UART2 tx (rx stays 0 — it only transmits); its FAIL/PASS verdict is rover-oriented, read the numbers.

Field-debug decision tree (symptom → cause, all previously seen)

1. Rover NO RTK never turns off, outdoors
   • Rover XBEE>GPS dark → no radio bytes: XBee antenna? radio ID drifted? → xbee_at.py, expect ID=1174.
   • XBEE>GPS blinking but stuck FLOAT → check base actually completed survey-in (fix=TIME in Pi log, or uart2 tx ≫ 12 B/s), then rover antenna sky view/multipath.
2. "Base LED blinks so it must be broadcasting" — NO. GPS>XBEE blinks from power-on carrying only static RTCM 1230 (measured with 0 satellites). Only survey-in completion starts 1005+MSM. LED-level proof of full corrections = a rover reaching FLOAT/FIX. Byte-level proof: UART2 tx rate — 12 B/s = 1230-only, ≫ 50 B/s = broadcasting.
3. Board absent from USB — before suspecting damage: try another cable (a dead/charge-only cable produced a false "damaged board" scare, 07-31), then the other USB jack (each jack has an independent bridge).
4. Serial reads return EPERM (macOS, this repo's home Mac) — known TCC bug on /Volumes/USRP01, NOT a device problem: memory usrp01-eperm-see-tcc-bug. Fix: sudo tccutil reset SystemPolicyRemovableVolumes com.apple.Terminal, touch the volume.
5. Base on a Raspberry Pi flaps on/off USB — undervoltage. dmesg shows Undervoltage detected! + over-current change + re-enumeration; the XLR's 1 W PA spikes through the Pi's USB rail. Fix: wall adapter into the board's second USB port (both powered simultaneously is fine), real 5 V/3 A Pi PSU. Also: ModemManager fights ttyACM ports — disabled on the beacon Pi 08-03; check systemctl is-active ModemManager on any new Pi.
6. Survey-in never completes indoors — expected physics, not a bug: meanAcc plateaus ≈ 15 m (measured 33.6→22.1→17.4→16.2→15.4 m over 256 s). The 20 m gate accommodates it. Writing ANY TMODE key restarts the running survey.
7. xbee_at.py can't enter command mode on the BASE — streaming RTCM breaks the +++ 1-s guard. The working in-place recipe (verified 2026-08-03, base PL read): connect BOTH base USB ports, then over the GPS port set CFG_UART2_ENABLED (0x10530005) = 0 RAM-only → line goes truly quiet → AT works on the XBEE port → read/set → restore =1. ⚠️ Setting the three CFG_UART2OUTPROT_* keys to 0 does NOT silence the line (ACKs, reads back 0, but bytes keep flowing — rover kept DGNSS for 5+ min); only UART2_ENABLED=0 actually stops TX. The board's XBEE RESET button + serial-break recovery was not needed.
8. Two processes on one serial port corrupt each other's frames — stop the Pi logger (pkill -f base_status_logger; systemd restarts it in 5 s) before ad-hoc reads on the same device.
9. CFG-VALGET returns nothing — one invalid key rejects the WHOLE request (NAK). Poll known-good keys in small batches.
10. Radio module vs board config — radio params (ID etc.) live in the XBee's own flash and travel with the module; F9P params live in the board. Modules on ID=1174 swap freely between boards (proven 07-29).

Environment specifics

• Home Mac (this repo's origin): ports appear as /dev/cu.usbmodem* (F9P) and /dev/cu.usbserial-* (XBEE+POWER bridge). Beware the TCC EPERM gotcha (#4).
• Beacon Raspberry Pi (ddh-pi4-beacon in Yi's ~/.ssh/config, 192.168.3.1, user user; it is its own WiFi AP, NO internet — pip cannot install, which is why the Pi logger is stdlib-only): base F9P at /dev/ttyACM* (number moves on re-enumeration — the logger's --port auto rescans). Logger + unit file live in /home/user/base_monitor/, service base-monitor, log ~/base_monitor/base_status.log.
• Any fresh field PC: clone repo, pip install pyserial, use tools/. Linux device names: F9P = /dev/ttyACM*, XBee bridge = /dev/ttyUSB* (FTDI).

Handoff log

• 2026-08-05 (Claude): Dashboard gains a rolling per-class quality view (proposed in data/20260805-run-eval/RESULTS.md after the 153-flap session): --stats-window-sec (default 300) deque of epochs → compute_window_stats() (stdlib, sample-std n−1, dash cells when n<2) → window object in /api/state → dashboard table (FIX/FLOAT/DGNSS: %, mean N/E/D, horiz std, D std — means expose the inter-class bias the run eval measured) + badge (green = FIX ≥80% AND FIX horiz std <5 cm; red = DGNSS-only/no data; amber otherwise, incl. FLOAT-dominated — FIX-dominant-but-noisy deliberately shows amber). CSV/session-log/serial paths untouched; Tk GUI untouched (state append only). Unit-tested against hand-computed values + demo-mode /api/state verified. Committed locally only — NOT pushed, NOT yet on mac-03 (its monitor restart pends Yi's go). (on Yi's Mac, indoors) reached TIME mode under the new gate (meanAcc=16.9 m @ 300 s), UART2 TX 12 → 376 B/s; rover #2 on field PC ddh-mac-03 received 383 B/s, diffSoln=1, RELPOSNED producing (garbage D=+23 m indoors — expected multipath, carrSoln=0). mac-03 provisioned: stale repo updated via git bundle over scp (field Macs have NO internet — git pull /tmp/rtk.bundle main), pyserial installed by copying the pure-Python serial/ package into the user site-dir. Use the same recipe for other field PCs (ddh-mac-0* in Yi's ssh config).
• 2026-08-03 (Claude): Base survey gate rewritten 120s/2.5m → 300s/20m (flash, verified) after measuring the ~15 m indoor plateau. Pi logger deployed as auto-start systemd service with --port auto; ModemManager disabled on the Pi; undervoltage root-caused (needs wall adapter on base's 2nd USB — pending). LED semantics corrected in all docs (GPS>XBEE ≠ TIME mode). Fixed-coordinate mode rejected for the one-shot campaign (see note.md open items). Rover #1 link check still pending. This file created.
• 2026-07-28/29 (Claude): Fleet bring-up: 5/6 rovers verified end-to-end; all fresh XLRs shipped ID=1985 (5/5) → set to 1174; UART2 38400→115200 on all new boards; roster + runbook + tools written.

2026-08-14 — CSV contract with the GPS↔host link tooling (Claude)

relposned_monitor.py's session CSV columns 1–2 (host_time,iTOW) and column 7 (carrSoln) are now consumed by LRLocal-Mac-EnvSetup/rtk-011-make-gps-link.py (builds gps_host_link.csv per session; it parses those columns positionally and drops rows shorter than 7 fields), and rtk-010-gps-sync-once.sh steps the mini clock from this receiver once before the monitor starts (field-000-jobs hook). Don't rename/reorder CSV columns 1–2 or 7 without updating rtk-011 — new columns are safe only appended at the end. The same warning sits on SessionLogger.CSV_HEADER in the monitor, at the point of change.