m365crawl

Imported from ourostack/m365crawl (AGENTS.md). Install upstream with npx skills add ourostack/m365crawl. Copyright stays with the author.

m365crawl agent instructions

Development rules

• Run every Go command with GOWORK=off. The Makefile and CI already do this.
• Run make check before opening a PR. It runs tidy, format, vet, lint (golangci-lint, govulncheck, actionlint), unit tests with the race detector, and the e2e tests.
• Every change reaches main through a PR with green CI. Pin every GitHub Action by full commit SHA with a trailing # vX.Y.Z comment.
• Never commit real Teams data, in fixtures, tests, logs, PR text or commit messages. Fixtures are synthetic and generated by the fixture generator (regenerate testdata/teams-fixture with make fixture, which drives Microsoft Edge through Playwright in scripts/fixture; after regenerating, refresh the goldens with go test ./internal/indexeddb -run TestFixtureGolden -update, review the diff and commit the tree, PROVENANCE.json and expected/ together; the generator fails loudly unless the origin directory name and the snappy, blob and v21 envelopes are all present; testdata/ files are binary and never reformatted).
• Never write to Teams' storage. Never decode credential material: the deny-only rule and the denylist are described under "Credential denylist" below. Never read Teams tokens.
• Every function in internal/... is 100% statement-covered by go test ./... (unit tests; the e2e and acceptance tags do not count). make coverage (scripts/check-coverage.sh) enforces the Unix 100% gate in the CI coverage job. The Windows companion scripts/check-coverage.ps1 retains legacy Windows file-execution checks and invokes scripts/coverage for strict native mail coverage: SDK-generated executable block spans and statement weights must match the actual profile, with every required function/block at 100%. Missing profiles, files, functions, blocks and zero-hit blocks fail; no mail carve-outs apply. Native build selection includes Windows OWA providers. Write tests that assert behavior, with synthetic data only. Prefer injecting a failure (an interface or a function variable) over listing a function.
• Carve-outs live only in coverage/allow/<package>.txt (one file per Go package, for example internal-leveldb.txt), one entry per line: <path relative to the repo root>:<FuncName> <reason>, the name exactly as go tool cover -func prints it. List a function only when its uncovered statements are unreachable from a deterministic test without unreasonable contortion (OS or platform glue, a syscall error that cannot be forced portably); "it would take effort" is not a reason. A stale entry (the function is already 100% or gone) fails the gate, so the list only shrinks. See coverage/allow/README.md. COVERAGE_PACKAGES narrows the check for local runs.
• Signal handling lives only in cli.Main. Lower layers take a context.Context and clean up when it is cancelled. The first SIGINT or SIGTERM cancels the context; a second one exits immediately with status 130 (exitForced), which can leave a snapshot directory in TMPDIR; a later sync sweeps snapshots older than an hour.
• Tests that rely on chmod 000 must skip when os.Geteuid() == 0.
• Do not add AI attribution anywhere: no Co-Authored-By trailers, no "Generated with" lines, no AI credit in commits, PRs, code comments or docs.
• Use conventional commit messages (feat:, fix:, ci:, chore:).
• Record a user-visible change in a changelog fragment, changes/<short-slug>.md (format in changes/README.md), never by editing CHANGELOG.md; CI refuses lines under ## [Unreleased]. A release assembles the fragments with scripts/changelog-assemble.sh X.Y.Z YYYY-MM-DD (see docs/releasing.md).

Credential denylist

• The rule is deny-only (operator ruling, 2026-10-02): agents get open access to all Teams data except credentials. internal/teamsdesktop/deny.go holds the denylist (Denied): a database or object store name that starts with Teams:auth or contains auth, token, credential, secret, cookie, msal, oneauth, key-store, keystore, keyval, session, key-value, keyring, crypto, encrypt, pkce, bearer, password, refresh or adal, or has aad, keys, jwt or e2ee as a whole -/_/:/.//-delimited or camelCase segment, ignoring case, is never opened for values, decoded, listed or logged. Only the counts denied_database and denied_store appear. It over-denies on purpose.
• internal/teamsdesktop/read.go holds the three typed managers (replychain-manager, conversation-manager, activity-manager), which have mappers. internal/teamsdesktop/generic.go (ReadGeneric) reads every object store that is not denied and not consumed by a typed mapper (typed databases' other stores included) into the records table, in memory-bounded batches (a census first, then one open per batch that keeps only that batch's stores by id, each origin dropped and the heap collected before the next). Values are scrubbed (scrub.go: JWTs, whole values of access_token-style keys (any type, including V8 Map pairs), Bearer strings, sig= parameters, credentials inside stringified JSON and name/value sibling fields, in values and record keys (a sync also clears rows archived under a key that scrubs differently), become [redacted], counted as redacted in the sync report), and each sync clears (PurgeDenied) archived rows whose database or store name the denylist now matches: the one place the archive deletes content.
• Changing the denylist is a design change: edit deny.go, its tests, SPEC.md section 2 and the fixture's credential decoys in the same PR. Weakening it (removing a term) needs a review of the real cache's database names first, on a machine that holds one. TestRealNoAuthDecoded asserts on a real cache that no denied database yields records.
• Adding a typed mapper for a generic database is a normal change: add the mapper, a fixture record, tests and a SPEC.md row; the database then leaves records for its own table. The calendar is the exception: it is a derived projection of records, and its stores (teamscal.ClaimedStores) stay in records and in stores. store.DeriveCalendar reads them back from records in the sync's transaction (internal/store/calendar.go).
• Raise teamsdesktop.RulesVersion when you change how deny.go or scrub.go apply their rules (a change to a table in ruleSet is covered by RulesStamp on its own), and teamscal.MapperVersion when a mapper change alters derived calendar data: the next sync then rebuilds the calendar tables from records under the new rules, so a derived copy does not outlive them.

Archive rules

• Never delete archived rows because they left Teams' cache. deleted_at (and records.removed_at) is a sticky tombstone (SPEC.md section 3.2). No DELETE against messages, conversations, people, activity or records.
• Never run VACUUM on the archive, in code or by hand. message_fts and conversation_fts reuse the rowid of the messages and conversations rows they index (both are rowid tables with no explicit integer primary key), and SQLite may renumber such rowids on VACUUM, which would detach every index entry from its row. The constraint is the rowid link, not FTS itself; if you add an explicit INTEGER PRIMARY KEY to those tables the constraint changes, so update store/schema.go, rederive.go and SPEC.md section 3.3 together.
• A change to a mapper that alters a derived field (text, sender name, conversation display name) must raise store.DerivationVersion and teach Rederive to recompute the field from raw_json. A change that alters what a decode produces must raise teamsdesktop.DecoderVersion so the next sync re-reads an unchanged cache. The same number keys the read memory (SPEC.md section 4.1): a sync skips a record whose stored bytes are unchanged only under the same teamsdesktop.MemoSignature, which covers DecoderVersion, store.DerivationVersion and every rule table of Scrub and Denied (they live in ruleSet, internal/teamsdesktop/rules.go; add a rule there, not in scrub.go or deny.go). A change to the code that applies the rules, or to a mapper, is not visible to the tables and still needs DecoderVersion raised, or records the mapper now reads differently are skipped. sync --full-read reads every record in full and must leave the archive unchanged.
• The calendar tables follow the same rules: no DELETE, no VACUUM; "gone" is a sticky removed_at and a recap item a newer copy drops is superseded_at. The archive schema is version 19 (store.SchemaVersion), distinct from derivation version 2 (store.DerivationVersion); internal/calendar's SchemaDDL is part of store's schema and the calendar tables are created once, there. Higher schemas are refused on reads and writes; migration is directional, with no downgrade or old-binary/future-archive promise (SPEC.md section 10).
• The mail tables (internal/store/mail_schema.go) follow the same rules: no DELETE of a mail_messages row and no VACUUM, because mail_fts, mail_recipients and mail_attachments use the message's implicit rowid. "Gone" and "evicted" are sticky marks set only after two trusted reads of different store copies miss a message (CommitMail). A change to outlookmail.HTMLText or store.NormalizeSubject raises store.MailTextVersion and Rederive recomputes the rows below it.
• The Outlook source is read-only and on by default (it reads the default directory when it is there; --outlook-root none or M365CRAWL_OUTLOOK_ROOT=none turns it off, and an explicit --teams-root without an explicit Outlook root keeps it off so tests stay hermetic; M365CRAWL_OUTLOOK=1 is accepted and does nothing). A default Outlook must never fail or slow a Teams sync (syncer.Options.OutlookImplicit). A profile is linked to a Teams account by address only when an address signed in to the profile equals the Teams account's own address; the operator's explicit link or none always wins. Any test that runs the CLI must give its roots (or a home directory of its own), or it will read the machine's real Teams and Outlook data. The pinned classes, tags and fields in internal/outlookcal and docs/outlook-store.md change together, and a mapper change that alters derived data raises outlookcal.MapperVersion. Never print a string from a real Outlook store in tests, logs, docs or PRs; fixtures are in testdata/outlook-fixture.
• Record schema and field changes in an unversioned changelog fragment in changes/. Within 1.x preserve the documented commands, flags/defaults, machine field names/types/null/omission semantics, streams/errors/exits, published SQL columns and scoped identities (SPEC.md section 10). Compatible additions preserve existing behavior; incompatible public changes require a new major release. A changelog entry alone never authorizes a 1.x break. An internal schema migration is not automatically a public API break or a promise that older binaries can read newer archives.

Documentation

• SPEC.md is normative. Change behavior and SPEC.md in the same PR, and add every new error code to its error table.
• docs/commands.md reproduces m365crawl --help and each m365crawl <command> --help. When a flag or help text changes, update it and check it against the binary.
• .agents/skills/m365crawl/SKILL.md is what a fresh agent reads first. Keep it short and consistent with SPEC.md; keep its platform guidance honest (macOS + Windows supported, Homebrew macOS-only, Windows assets intentionally unsigned); example JSON in docs comes from the committed fixture, never from a real cache.
• Never put real Teams data in docs, examples, issues or PRs.

Real-cache acceptance tests

• M365CRAWL_REAL_CACHE=1 make acceptance (or GOWORK=off go test -tags acceptance ./acceptance/... when you need the direct Go entrypoint) runs acceptance/ against the live Teams cache on the current supported host. It needs Full Disk Access on macOS, a live signed-in Teams cache on Windows, Node 22, and python3 with two read-only reference clones: ~/code/_refs/ccl_chromium_reader and ~/code/_refs/ccl_simplesnappy (override with CCL_READER, CCL_SNAPPY; M365CRAWL_PYTHON picks the interpreter). ccl_count.py stands in empty brotli and zstd modules, so no pip install is needed. CI never runs it, and without M365CRAWL_REAL_CACHE=1 the real-cache tests skip.
• TestRealDifferential decodes every allowlisted typed record twice, with v8.Deserialize and with Node's v8.deserialize (acceptance/diff.mjs), and compares canonical JSON. TestRealVolumes and TestRealConversationAccounting compare record and message counts with ccl_count.py on the same snapshot; the reference counts only the newest version of each record, so superseded versions and deletions are excluded by design.
• TestRealRecency checks that the newest archived message is within an hour of the newest conversation activity, and TestRealBlobsResolve asserts that a real sync reports zero blob_missing omissions. TestRealNoAuthDecoded also scans the archive for anything shaped like a bearer token.
• The tests log counts, timings and field paths only. Never add a log line, failure message or comment that prints a name, id or content from the real cache.
• TestRealOutlookCost reports resource magnitudes rather than imposing fixed cache-size, time or RSS pass/fail limits. Capture, sync, measurement-protocol and source-safety failures still fail; unavailable platform metrics are explicit, never reported as measured zero. Production resource bounds, cancellation and test timeouts are unchanged.
• M365CRAWL_REAL_CACHE=1 make acceptance-calendar (or -run 'TestRealCalendar|TestRealOutlook' on the direct Go entrypoint) runs only the calendar checks: acceptance/calendar_test.go (the Teams calendar, recap and recording checks) and acceptance/outlook_test.go (the Outlook gating checks). The Teams root is M365CRAWL_TEAMS_ROOT (default: the installed Teams); the Outlook checks need M365CRAWL_OUTLOOK_ROOT, a directory laid out as <root>/<profile>/HxStore.hxd, and skip with a message without it. M365CRAWL_ACCOUNT and M365CRAWL_OUTLOOK_PROFILE choose the Teams account and Outlook profile when there are several, and M365CRAWL_ACCEPTANCE_WATCH=1 adds the ten-minute watch simulation of the cost check, and M365CRAWL_ACCEPTANCE_PROFILE=<dir> writes a CPU profile of the first Outlook sync and a heap profile (inuse and alloc) taken at its live-heap peak (function names and sizes only). TestRealOutlookDeletionProbe (acceptance/outlook_deletion_test.go) pins the deletion experiment: it syncs copies of one store taken before a deletion of test appointments, optionally after it and before compaction, and after compaction (M365CRAWL_OUTLOOK_PROBE_BEFORE, _LINGER, _AFTER, each laid out as <root>/<profile>/HxStore.hxd, and M365CRAWL_OUTLOOK_PROBE_IDS, the lower-case id suffixes of the deleted appointments) (plus M365CRAWL_OUTLOOK_PROBE_CONFIRM, a later copy that also lacks them) and expects exactly those events marked gone, only by the last copy; it needs no Teams cache and skips without the variables. Every threshold lives in calendarThresholds in acceptance/calendar_support_test.go. The checks only read the sources (the Outlook store is copied by the sync, never opened for writing), log counts, field names, zone labels, durations and sizes only, and never a subject, name, address, URL, id or body.

Test hooks

• M365CRAWL_TEST_PAUSE_AFTER_SNAPSHOT=<duration>: when set, the syncer prints m365crawl-test: paused after snapshot to stderr and then sleeps that long after the snapshot is taken. Used by the e2e tests, which signal the process once the marker appears, and by a few unit tests that cancel a sync during the pause. Never set it outside tests.
• M365CRAWL_TEST_PAUSE_IGNORES_CANCEL=1: makes that pause ignore cancellation, so a graceful stop cannot finish and the e2e test can prove that a second SIGINT or SIGTERM force-quits with exit 130.