sherlock

Imported from benpate/sherlock (AGENTS.md). Install upstream with npx skills add benpate/sherlock. Copyright stays with the author.

Sherlock — Notes for AI Agents

• Load never hard-fails on a single source; it tries many and merges. Each metadata format is attempted in turn and merged into one ActivityStreams document without overwriting values already found. A fetch or parse error from one source is swallowed so the others still run. Don't expect an error just because one format was absent or malformed.

• The subpackages are stacked client middlewares, and stacking ORDER is important. bridgyfed and tagspub rewrite identifiers into WebFinger handles, so they MUST sit above webfinger in the stack; webfinger resolves handles to URLs that activitypub then loads; tombstone substitutes a placeholder for Gone documents. Each subpackage's README states its own placement rule.

• Network access is SSRF-hardened by default, inherited from remote. AllowPrivateIPs defaults to FALSE, so private/loopback fetches are blocked and response sizes are capped. Self-hosted/LAN targets, and tests against httptest servers, must pass WithAllowPrivateIPs(true).

• Every fetch in the root package is built by Config.newTransaction. It is the single place that applies the User-Agent, the SSRF setting, and the caller's remote options, so a call site that reaches for remote.Get directly silently opts out of all three — which is exactly how the homepage-icon lookup came to run with neither the configured User-Agent nor AllowPrivateIPs. Libraries that build their own transaction (digit.Lookup) get Config.remoteOptions() instead, which carries the SSRF setting as a BeforeRequest hook.

• remote.Options is variadic, so the spread is important: remote.Options(options...). A streams.Client.Load carries its options as ...any, and []any satisfies any — so remote.Options(options) compiles, passes the whole slice as a single value, matches no Option, and returns an empty set. Every caller option is dropped with no error anywhere: the document still loads, just unmodified. activitypub.Client.Load shipped that way and dropped every per-call option reaching the terminal client in the stack, including the Accept header bridgyfed injects above it. TestLoad_PerCallRemoteOptionReachesRequest locks it. Any new middleware that filters the ...any bag needs the same spread, and the matching rule for forwarding a Load down the stack is in Emissary's tools/AGENTS.md.

• Config travels BY VALUE, so a callee cannot mutate it. Every load* method takes config Config, not a pointer. Writing to a field inside one of them changes a copy that is discarded on return — which is why MaximumRedirects-- after a recursive call left the redirect budget at its starting value forever, and let two pages that link to each other recurse until the stack gave out. Spend the budget BEFORE recursing, never after.

• DefaultValue is copied at option time, and the loaders write into the copy. WithDefaultValue clones the caller's map, because the RSS, JSON Feed, MicroFormats, and HTML loaders all write their findings into config.DefaultValue directly. Without the clone, a caller who reuses one map across Load calls gets the first document's fields leaking into the second. All four loaders seed from DefaultValue and then overwrite it with what they discover, so a caller's value survives only for keys the source did not supply.

• Identifier classification is strict and lives in uri. Whether a value "looks like" a URL or an @handle is decided by uri validation (real IANA TLDs, 2+ segments). When a test for that classification fails after a uri upgrade, the test assumption is usually what drifted — not the code.

• Untrusted-input parsers are fuzzed; keep new ones that way. The document and identifier parsers (OpenGraph, microformats, embedded JSON-LD, address/identifier classification) have Fuzz* coverage in fuzz_test.go. Regexes are static patterns (regexp.MustCompile) — no untrusted input is ever compiled into a regex. A new parser of remote bytes should arrive with its own fuzz target.

• The metadata engine (the metadata/ sub-package) never touches hannibal. Client.Metadata delegates to metadata.Get, which returns the oEmbed-adjacent metadata.Preview built by pure extractors merged in precedence order (AS → OG → Twitter → oEmbed → HTML). The layering is enforced by the package boundary: metadata has no hannibal import, and the future AS2 projection file lives in the ROOT package, importing metadata — never the reverse. Extractors return sparse partials (unexported — the engine's internals are deliberately not API) and NEVER write into a Preview — write-if-empty into a shared result is the old defect pattern this engine replaces. The streams.Document getters are live (they can fetch); extractors must stay pure over bytes already in hand. A few tiny helpers (isActivityStream, defaultHTTPS, iconSizesAsInt) are deliberately duplicated in metadata/utils.go rather than exported from the root, to keep the dependency pointing one way.

• A partial's fields are nullable values, and IsZero() is a trap. They are null.String / null.Object[T] (rosetta), never pointers, so nothing in the package dereferences — a null field reads back as its own zero value, which is why hasEmbedSignals can switch og.Kind.Object() with no presence test. merge must ask IsPresent(): IsZero() reads TRUE for a present-but-empty value and would silently demote a source that did speak. The one deliberate exception is firstString, which wants a usable value and so does use IsZero(). Groups reach the Preview as copies, so the icon backfill can never write back into the partial that won.

• Precedence is argument order, plus ONE named exception pick. merge() call order in metadata.extract IS the global precedence (AS → OG → Twitter → oEmbed → HTML); the only deviation is the URL pick (AS id → rel=canonical → og:url, same-origin required), which merge never touches. Groups (Thumbnail/Embed/Provider/Authors) fill whole from one source with validity floors — never mix one source's URL with another's dimensions. Changing precedence means editing metadata.extract, nothing else.

• Embed has NO exception pick — don't reintroduce one. Until 2026-08 extract() ended with result.Embed = firstValidEmbed(oe, tw, og), which unconditionally overwrote whatever merge had chosen. That made TWO competing definitions of embed precedence, and the one that looked authoritative (merge order) was inert — reordering the merge() calls changed nothing. The override is gone: an og:video player now beats an oEmbed player, because OG merges first. The tradeoff is deliberate and important: og:video is ambiguous (file vs. player), so finalizeOpenGraphEmbed classifies it and validEmbed rejects a file posing as an iframe. TestMetadata_EmbedPrecedence_* in get_test.go locks the rule end to end — five of those tests fail if the override comes back.

• The oEmbed endpoint call is lazy and must stay that way. (result.Embed == nil && hasEmbedSignals(...)) || oembedCouldHelp(result) gates the only extra network hop (both helpers live in extract-oembed.go). The result.Embed == nil guard matters: precedence is call order, so an embed already merged from OG or Twitter has won, and fetching oEmbed to improve on it would buy nothing. oembedCouldHelp deliberately does NOT test Embed (every preview lacks an embed before oEmbed runs — testing it would make the gate always fire) or Description/Language (the oEmbed spec has neither). It replaced a fieldSet bitmask + Preview.couldImprove in 2026-08: the mask had one caller and one value, and half its bits were never set — don't reintroduce a capability calculus for a single gated source. Provider embed HTML enters the model only through oembed.Embed() classification. Whether non-extractable markup becomes a sandboxed embed is the operator's AllowSandboxEmbeds option (default FALSE = no embed) — a uniform UX/product decision, NOT a trust tier keyed on how the endpoint was found. Don't reintroduce registry-vs-discovery provenance: the sandbox plan runs in an opaque-origin iframe and is MORE contained than the plain iframe plan, so gating it on provenance was backwards, and the oembed library deliberately no longer reports provenance.

Known defects, reproduced and filed

Four defects are reported and unfixed. Read the reports before treating any of them as a new finding.

• Load fails on EVERY HTML page (BUG-135). loadDocument returns early when loadDocument_ActivityStream errors, and remote refuses to decode text/html into a map, so the OpenGraph, microformats, and JSON-LD extraction below it is unreachable — not merely the DefaultValue fallback. This blocks the RSS following restore.

• Icon discovery picks the WORST icon (BUG-134). Both halves of the comparator are inverted: sortImageLinks sorts ascending and the caller takes icons[0], so the smallest wins on size, and an unrecognized media type (score 0) beats webp (256) on the tie-break.

• The microformats published-date default never fires (BUG-133). The guard compares an any against the untyped constant 0. Unset reads as nil and a parsed date is int64, so only a literal int(0) would ever match — the branch is dead in every reachable case and the property is left unset.

• A link walk amplifies fetches without bound (BUG-136). The redirect budget bounds DEPTH but not total work; one Load was measured issuing 39,062 fetches at the default budget of 6.

A related test trap, already fixed: bridgyfed.TestLooksLikeBluesky asserted that not-enough-segments.net was invalid. It has two segments and a real TLD, so the stricter uri.ValidateHostname correctly accepts it — the test encoded an obsolete assumption, and the function was right. Fix a case like that in the test, not the function.