covid-mobility

Imported from danylaksono/covid-mobility (AGENTS.md). Install upstream with npx skills add danylaksono/covid-mobility. Copyright stays with the author.

AGENTS.md — UGM Mobility Data Analysis (Yogyakarta, COVID-19)

This file is the single source of truth for working in this repository — for human contributors and AI agents. Read it before making changes.

1. What this repo is

A COVID-19 mobility study using anonymous mobile-device GPS data from Daerah Istimewa Yogyakarta (DIY), Indonesia.

• Mobility data: GPS pings (maid device id, lat/lon, unix timestamp).
• People graph: device-level profiles (sex, activity intensity, admin geography place1..3).
• Prepared window (in data/parquet): 23 Oct 2021 → 7 Jun 2022, all nine months ingested from raw CSVs (~292.2M pings, ~4.3M devices).
• One data/parquet/mobility_<MonthYear>.parquet per month (multi-part months combined), plus a combined data/parquet/mobility.parquet.

2. Architecture — separation of concerns

mobility/
├── AGENTS.md                        # this file
├── research_plan.md                 # analysis plan (COVID themes + viz roadmap)
├── README.md                        # human quickstart
├── FINDINGS.md                      # analysis results (keep updated)
├── requirements.txt                 # pinned deps (duckdb, h3, ...)
├── pyproject.toml                   # src layout; `pip install -e .`
├── src/mobility/                    # ★ reusable processing package (duckdb + h3)
│   ├── config.py                    #   paths + constants (H3 res, CSV conventions)
│   ├── io.py                        #   CSV→parquet via DuckDB; loaders
│   ├── clean.py                     #   cleaning rules (artifact rows, \N, timestamps)
│   ├── temporal.py                  #   DuckDB aggregations (daily/hourly/diurnal)
│   ├── spatial.py                   #   H3 grid aggregation + centroids/boundaries
│   ├── homes.py                     #   home-cell detection (baseline nights)
│   ├── metrics.py                   #   per-device-day metrics (gyration, trips, stay-at-home)
│   ├── contacts.py                  #   co-location / meeting index (exposure proxy)
│   └── od.py                        #   origin–destination flows
├── scripts/                         # ★ one-shot pipeline steps (run via CLI)
│   ├── process_mobility.py          #   raw CSVs -> cleaned parquet (duckdb)
│   ├── prepare_spatial.py           #   H3 grids + OD + static maps -> data/processed
│   ├── compute_metrics.py           #   homes + per-device-day metrics -> data/processed
│   ├── compute_contacts.py          #   meeting index + crowding -> data/processed
│   ├── profile_full.py              #   whole-dataset profile (date-chunked, RAM-bounded)
│   ├── run_all.py                   #   run the whole pipeline in one go
│   ├── plot_summary.py              #   static summary plots -> outputs/plots
│   ├── plot_traces.py               #   per-device movement traces -> outputs/traces
│   └── explore_data.py              #   sanity-check any dataset
├── notebooks/                       # ★ thin notebooks: load prepared data + plot ONLY
│   └── UGM_Mobility_Data_Analysis.ipynb
├── web/                             # interactive trace inspector (MapLibre GL JS)
│   ├── index.html  style.css  main.js
│   └── data/*.geojson               #   generated by scripts/export_traces_web.py
├── archive/                         # old versioned notebooks (read-only)
├── data/
│   ├── raw/                         # source CSVs (drop new months here)
│   ├── parquet/                     # cleaned parquet from process_mobility.py
│   ├── processed/                   # H3 grids, OD flows, homes, metrics
│   └── legacy/                      # pre-processed Oct-2021 parquet (read-only)
└── docs/
    ├── DATA_STRUCTURE.md            # data dictionary / metadata
    └── WORKFLOW.md                  # reproduce & extend instructions

Golden rule: notebooks never do heavy processing. Data processing lives in src/mobility + scripts. Notebooks load prepared Parquet and make static plots only. No lonboard / interactive maps in notebooks (static PNGs suffice for now).

3. Data processing stack

• DuckDB for CSV ingestion & aggregations — streaming, parallel, low memory. Full 12M+ rows are never materialised in Python for simple aggregates (src/mobility/temporal.py queries Parquet directly).
• Uber H3 for spatial aggregation (src/mobility/spatial.py).
  • ⚠️ The installed h3==4.5.0 (Windows / py3.14) exposes a scalar, string API only. latlng_to_cell is wrapped with np.frompyfunc to vectorise (~1.2M pts/s). Cell ids are strings like '888d8cb95dfffff' (there is no cell_to_string / string_to_cell in this build).
• Results are cached as Parquet under data/processed/ so notebooks and future analyses are fast.

4. Known data quirks (MUST handle — already encoded in mobility.clean)

5. Conventions

• Path constants live in mobility.config — never hard-code paths in scripts/notebooks.
• Timestamps: parse with pd.to_datetime(int, unit='s'); store cleaned data with datetime (TIMESTAMP), date (DATE), hour columns.
• H3 cells stored as strings in Parquet (compact, sortable).
• Aggregations prefer DuckDB-over-Parquet over loading into pandas.
• When a new quirk or schema change is found, update clean.py, docs/DATA_STRUCTURE.md, and this file.

6. How to run

.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
pip install -e .                     # makes `import mobility` work everywhere

python scripts/run_all.py --sample 500000   # or run the whole pipeline in one go
python scripts/plot_summary.py --out outputs/plots   # static summary PNGs
python scripts/plot_traces.py --day 2021-10-30 --n 4  # per-device movement traces (needs data/geo boundaries)
python scripts/explore_data.py --quick          # sanity-check
python scripts/process_mobility.py --mobility "data/raw/*.csv" --people data/raw/people_graph.csv --out data/parquet
python scripts/prepare_spatial.py --out data/processed --plot outputs/maps
python scripts/compute_metrics.py --out data/processed --res 8   # homes + device-day metrics
python scripts/compute_contacts.py --out data/processed --res 8 --bucket 1h   # meeting index
code notebooks/UGM_Mobility_Data_Analysis.ipynb  # run cells top-to-bottom

7. Roadmap (where this is heading)

1. Ingest more data (Oct 2021 → May 2022): drop CSVs in data/raw/, run scripts/process_mobility.py. Extend prepare_spatial.py if new months need re-aggregation.
2. Spatial / OD analysis (data-prep is the priority; visualization later):
   • H3 grid density (done, src/mobility/spatial.py).
   • Per-device-day metrics + homes (done, src/mobility/metrics.py + homes.py, scripts/compute_metrics.py).
   • OD matrices (done, src/mobility/od.py: device-day origin/dest + flows,
     • flow distance, corridor change vs baseline, district-level OD, net flows — scripts/prepare_spatial.py --people ...).
   • Contact / meeting index (done, src/mobility/contacts.py + scripts/compute_contacts.py).
   • Spatial concentration per day (done, spatial.concentration: Gini/HHI).
   • Road network integration (planned): snap OD trips to a road graph (e.g. OSM), estimate route distances/travel times.
3. COVID policy overlay: join daily mobility against the PPKM restriction timeline for Yogyakarta.
4. People-graph joins: mobility × demographics (sex/intensity/regency) for behavioural comparisons.
5. Visualization (later): glyph library for per-device and OD data; static small-multiple / OD maps. See research_plan.md for the full analysis plan.

8. Agent do's & don'ts

• Do put processing in src/mobility / scripts; keep notebooks thin.
• Do reuse mobility.clean / mobility.io for any new file.
• Do cache expensive spatial/OD/metrics/homes outputs to data/processed/.
• Do follow research_plan.md when adding COVID analysis themes.
• Do append dated sections to FINDINGS.md instead of overwriting.
• Don't load 260M+ rows into pandas when a DuckDB query suffices.
• Don't add lonboard/leafmap to the core pipeline (optional extras only).
• Don't hard-code absolute paths; use mobility.config.
• Don't delete archive/ — it holds versioned work.