glaucoma-thesis

Imported from Tqhuyen/glaucoma-thesis (AGENTS.md). Install upstream with npx skills add Tqhuyen/glaucoma-thesis. Copyright stays with the author.

AGENTS.md

Working guide for AI agents contributing to this repository. It documents the criteria (what "done" / "correct" means) and the flows (how data, training, logging, and experiments work) so agents don't guess.

Project in one paragraph

Master thesis: 3D CNN for glaucoma detection from OCT volumes. One model-agnostic training pipeline that runs unchanged on Colab, vast.ai, a single GPU box, or multi-node SLURM. The thesis dataset is Harvard-GF (3D OCT volumes, 200³ uint8). Storage resolution is config-driven for notebook experiments (STORE_RES: 200/128/96, …); the main pipeline config uses raw 200³.

Criteria (what counts as "correct")

Code style

• Line length 120; ruff enforced (see pyproject.toml):
  • make lint = ruff check pipeline models tests + ruff format --check
  • Never edit archived notebooks for lint — pre-existing notebook lint noise is accepted (CI only lints pipeline models tests, not notebooks/).
• No code comments unless asked, except mandatory notebook config-group headings in docs/notebook-conventions.en.md; follow existing patterns (registry pattern, config-driven, rank-0-only logging).
• Pipeline forward() contract: return {"logits": tensor} (or HF-style object with .loss).
• Pipeline DataLoader must return (x, y) tuples or dicts with "labels".
• Standalone multiview notebooks may use raw tensor logits and (x, views, y) batches when their shared trainer expects those contracts; do not change the pipeline contract.

Performance

• Default to the fastest correct implementation: AMP (+ GradScaler), grad accumulation, lazy/cached loaders, pin_memory + non_blocking on CUDA, vectorized ops, cached preprocessing.
• Bound heavy analyses (MI/surrogate/X-AI/metrics) by subset/steps; never add GPU/CPU work that is not needed or logged.
• Pick batch size against the real config+data with ft.find_batch_size and a VRAM budget; keep the effective batch constant via grad-accum and reuse the saved batch on resume.
• Measure sec/epoch and peak VRAM before/after optimizing; keep the CPU smoke path fast.

Tests

• make test / pytest tests -q must pass (9 tests).
• Config schema validates in <1s (pipeline/config.py): presence + type + range. Optional keys validated only if present (old configs stay valid).

Config as contract

• configs/glaucoma.yaml is the only glaucoma config (raw 200³). Never re-add glaucoma_96.yaml / glaucoma_128.yaml.
• Every config change must keep validate_config(cfg) green (add optional keys to _OPTIONAL_SCHEMA, not the required base schema, unless truly required).

Model convergence (the "4 traps" checklist)

When a 3D CNN on 200³ OCT doesn't converge, check in this order:

1. Resolution destruction — pooling must NOT crush the thin RNFL layer. Fix: anisotropic pooling stride (2,2,1) keeps B-scan depth resolution; AdaptiveAvgPool3d(1) only at the very end. See models/glaucoma/model.py.
2. BatchNorm with tiny batch — batch_size 2–4 (VRAM) breaks BatchNorm. Fix: GroupNorm (norm: "group", norm_groups: 8). Gradient accumulation does NOT help BN (it only affects effective batch).
3. Pixel intensity — data is uint8 0–255; /255.0 (minmax) is correct. For robustness use normalize: "robust" (percentile 1/99 clip + z-score). Always verify min/max before training.
4. Loss/output mismatch — raw logits (nn.Linear(64, num_classes)) must pair with CrossEntropyLoss (never sigmoid+BCELoss with raw logits, never MSE).

Overfit sanity gate (mandatory before blaming data): make sanity (or python -m pipeline.train --cfg configs/glaucoma.yaml --sanity) overfits 10 samples, 100 epochs, dropout=0/wd=0/warmup=0, no eval/save.

• Loss → ~0 (e.g. 1e-3) and train acc → 1.0 ⇒ code is correct; problem is architecture/LR/data size.
• Loss stuck ⇒ bug is in code/arch/data, not the dataset.

Commands

make setup    # pip install -e ".[dev,glaucoma,nlp]"
make lint     # ruff check pipeline models tests
make test     # pytest tests -q
make smoke    # quick end-to-end validation (CPU)
make sanity   # overfit-10-samples convergence gate
make train    # full train on raw 200³ (configs/glaucoma.yaml)
make ddp      # single-node multi-GPU
make tb       # TensorBoard live at localhost:6006
make plot     # re-render training_curves.png from metrics.jsonl

Flows

Data flow

harvardairobotics/Harvard-GF (HF, per-scan .npz 'oct_bscans' 200³ uint8)
  → scripts/harvard_oct_processor.py   (streams + uploads consolidated .npy)
  → glaucoma_all/{Training,Validation,Test}_{volumes,labels}.npy
  → models/glaucoma/data.py            (GlaucomaNpyDataset, mmap)
      normalize: minmax (default) | robust | none
  → pipeline/train.py

• Storage resolution is config-driven (STORE_RES: 200/128/96, …); model input (RES3D/RES2D) is declared separately and resized on the fly. Never hardcode 200.
• Always download from HF with credentials (HF_TOKEN from .env/Colab Secret, passed as token=), and only the splits/patterns actually used (allow_patterns / per-file hf_hub_download) — never the whole repo; prefer HF_XET_HIGH_PERFORMANCE=1 for max speed (the old HF_HUB_ENABLE_HF_TRANSFER is deprecated).
• Cache/reuse processed data (views, denoise, heavy augmentation); per case decide whether it is worth uploading to a versioned HF dataset repo (method + params
  • implementation hash) so later runs skip recompute — ask before uploading.
• Persisted Bilateral-96 storage (approved exception to "never persist downsampled"): private HF dataset tqhuyen/harvard-oct-glaucoma-200-bilateral-96 rev 7b528ec4c72e63bfc3394d185911e75c3f323813 (superseded first commit 4372a41f… was empty). Files per split: {split}_volumes_dn96.npy (uint8, 96³), {split}_labels.npy, {split}_complete.json, plus manifest.json (shapes + SHA256 + provenance). It is derived from the Bilateral-200 export (tqhuyen/harvard-oct-glaucoma-200-bilateral rev 47632c96b206707fd6423ee5b4da159069f63eaf) by float32 /255 → trilinear align_corners=False → round/clip uint8, and is bit-identical to FinalDataset's on-the-fly resize (verified). Use it only to feed the 3D branch; 2D en-face views must still be projected from the 200-cubed source. Rebuild/re-upload with python scripts/make_bilateral_96.py [--upload] and HF_XET_HIGH_PERFORMANCE=1; download selectively with HF_TOKEN + allow_patterns.
• mmap stored volumes; on Windows use num_workers=0 (mmap + multiprocessing can segfault).

Training flow (pipeline/train.py)

config.yaml → validate_config (fast fail) → build loaders → build model
  → DDP wrap → AdamW + warmup/cosine + AMP GradScaler
  → loop: compute_loss → scaler.backward → clip → step (grad_accum)
  → live train/acc every log_every_steps
  → val eval every eval_every_steps → best-model save + early stop
  → test eval every eval_test_every_steps (live) + final held-out test
  → render_and_sync(): PNG curves + metrics.jsonl + best_model.pt → Drive
  → run_manifest.json (reproducibility)

• --smoke / --sanity / --resume profiles applied via apply_profile(cfg, args).
• SIGTERM/SIGINT → atomic checkpoint → clean exit; --resume continues.
• Rank 0 exclusively: logging, checkpointing, test eval, plotting.

Logging flow (3 sinks + Drive)

• TensorBoard: live, outputs/<run>/tb (Colab inline / make tb).
• JSONL: outputs/<run>/metrics.jsonl — per split (train/, val/, test/): loss, acc, precision, recall, f1 (+ _pos = class-1/glaucoma-positive), plus sys/cpu_percent, sys/ram_used_gb, sys/ram_percent, sys/disk_free_gb, sys/gpu_* (when CUDA).
• W&B (MANDATORY — every run/experiment must log to it): project glaucoma-thesis.
  • Same keys as JSONL, logged live every log_every_steps + each eval.
  • GPU/CPU/RAM/disk auto-tracked by wandb's System panel; sys/* metrics also logged explicitly.
  • Key from .env (WANDB_API_KEY) via load_env_file() in pipeline/utils.py — always call load_env_file() before wandb.init().
  • .env is gitignored; never commit real keys.
• Drive sync (MANDATORY for EVERY artifact) — every image/figure/model an agent produces must also be saved to Google Drive under the repo's Drive root /content/drive/MyDrive/MasterBKDN/Thesis/ (this is what every notebook uses — see SAVE_DIR/RESULTS_DRIVE/output_dir in notebooks/*.ipynb), typically in a per-experiment subfolder: training curves/metrics/checkpoints go under logging.drive_sync_dir (e.g. .../Thesis/sota_200, .../Thesis/multiview, .../Thesis/denoise_sweep), notebook/script figures go under .../Thesis/<experiment>_figures mirroring figures/** (denoise_compare_*.png, denoise_metrics_*.csv, meta JSON), report views, etc. "Saved locally or to git only" is NOT done — copy to Drive too.

W&B mandatory process (do this in EVERY experiment)

1. Set the key — .env WANDB_API_KEY=wandb_xxx (repo root, gitignored) or Colab Secrets WANDB_API_KEY. Run python -c "from pipeline.utils import load_env_file; load_env_file()" or just start any run — the pipeline auto-loads it.
2. Pipeline runs (pipeline/train.py): logging.wandb: true in the config is already on. Every run creates a run named run_name in project glaucoma-thesis and streams train/val/test loss/acc/precision/recall/f1 + sys/* live.
3. Notebooks (notebooks/*.ipynb): every notebook MUST include the wandb mount-cell helper (init_wandb(run_name, config) + WANDB_RUN), init before training, run.log({...}, step=...) per epoch/step for train+val, log test at the end, then run.summary.update(...) + run.finish(). If a new notebook is added without this, it is NOT done.
4. Never disable or drop wandb to "save time" — it is the live experiment log.

Notebook conventions

• Agents MUST read docs/notebook-conventions.en.md before creating, editing, or reviewing any training/research notebook. Always use the English version as the canonical notebook specification, not the Vietnamese human-facing companion docs/notebook-conventions.md. Keep both language versions synchronized when changing conventions. The rules below are a summary; explicit experiment limits still apply.
• Mandatory pre-push real-run check (docs/notebook-conventions.en.md §3.15): before every commit/push, run smoke locally, then switch all *_SMOKE flags back to real run (unset or 0), remove smoke-only defaults, and set the real identity/stage switches. A pushed notebook/config must run the real experiment immediately; report the check with the push.
• Full cell structure + hard rules when creating a training/sweep notebook: docs/notebook-conventions.en.md — canonical example is notebooks/3d_glaucoma_multiview_sota_sweep_xai.ipynb.
• Data policy: authenticated selective HF download (token + allow_patterns), config-driven STORE_RES (200/128/96/…), reuse caches, and evaluate versioned HF upload for expensive processed data (denoise/views/augmentation).
• Notebooks are standalone Colab experiments (inline code, !pip/!git cells).
• Every figure/model a notebook produces must be saved to Drive: mount Drive, then copy figures/**, checkpoints, CSVs/meta to DRIVE_SYNC_DIR = /content/drive/MyDrive/MasterBKDN/Thesis/<experiment>_figures before run.finish() (guard with env override so headless runs don't fail on mount). Follow the exact paths used by existing notebooks (sota_200, multiview, denoise_sweep, 3dino_ft, ...).
• Variable shadowing trap: define storage resolution (STORE_RES=200, raw) and model-input resolution (MODEL_RES=112) as distinct names. Do NOT reuse one RESOLUTION var for both — cell 6 must not overwrite cell 4's value (this caused a real patch-embed assert bug).
• 3DINO-ViT needs input divisible by patch 16 → 112³ (7³ tokens). Resize on-the-fly in preprocess_volume (cast to float before F.interpolate — trilinear fails on uint8), never persist the downsampled data.
• Wandb in notebooks: init_wandb(run_name) helper (idempotent via WANDB_RUN is None guard), log probe + per-epoch train/val + test, then run.summary.update() + run.finish().

Repo layout (key files)

pipeline/train.py        training loop (DDP, AMP, preemption, test eval)
pipeline/config.py       schema validation (required + optional keys)
pipeline/metrics.py      TensorBoard + JSONL + wandb (load_env_file first!)
pipeline/plotting.py     render_curves() + sync_to_drive() + render_and_sync()
pipeline/utils.py        seeding, atomic save, load_env_file()
models/glaucoma/model.py Simple3DCNN: GroupNorm + (2,2,1) pool + residual
models/glaucoma/data.py  GlaucomaNpyDataset (minmax/robust/none)
configs/glaucoma.yaml    the ONLY glaucoma config (raw 200³)
notebooks/3d_glaucoma_3dino_experiment.ipynb  3DINO-ViT transfer + wandb
scripts/                 colab_setup, vast_setup, launch_ddp, slurm
docs/                    thesis docs (index in docs/README.md): datasets,
                         denoise comparison, multiview model, train platform