httrack

Imported from xroche/httrack (AGENTS.md). Install upstream with npx skills add xroche/httrack. Copyright stays with the author.

AGENTS.md — working in the HTTrack tree

Policy and PR etiquette live in CONTRIBUTING.md. This file is the operational checklist: toolchain, invariants, and how to ship a change.

Build & test

• Fresh clone first: git submodule update --init src/coucal
• ./bootstrap (regenerates configure via autoreconf; needs autoconf, automake, libtool), then bash configure && make -j"$(nproc)" && make check -j16. Always pass -j to make check: the suite runs under automake's parallel harness and each crawl test binds its own ephemeral-port server, so -j never contends and a multi-minute serial run drops to seconds. A new .test added to $(TESTS) is scheduled onto a free worker automatically; only a test slower than the current longest raises the floor. The right width is not a core count. Tests mostly sleep, waiting on a server trickle or httrack's own pacing, so an idle core covers a sleeping one. Measured on 4 cores, -j8 takes 245s and -j16 161s, so CI passes a flat 16 through the CHECK_JOBS variable in ci.yml. That includes macOS, because the test server raises its listen backlog (request_queue_size) so macOS/BSD don't drop connections under a parallel -c16 bigcrawl the way Python's default backlog of 5 did. Or run sh build.sh to do bootstrap + configure + make in one shot.
• configure globs TESTS from tests/[0-9]*_*.test, so a new test needs no registration, but an existing build dir keeps the list it was configured with: 231_test-names.test goes red until you reconfigure. It also compares the glob against git ls-files, so a test you forgot to git add fails there instead of quietly shrinking CI's suite. Name one outside the pattern and it never runs; tests/check-test-names.sh (also a CI lint) rejects that.
• A test that skips on Windows needs registering twice. The Windows job compares its skips against the written-out lists in tests/ci-windows-suite.sh, so an all-skipped suite cannot report green having tested nothing. A new skip_on_windows test reds libhttrack (x64, Release) with "skip set changed from expected" until its name is in BOTH the msys and the wsl2 list, with its reason in the comment above them.
• make check puts a RELATIVE src/ on PATH. A test that changes directory and then runs httrack by name finds the INSTALLED one, which fails in ways that look like the change under test. Resolve the binary to an absolute path before the first cd.
• make check prepends the build's src/ to PATH, but a hand-run .test does not — an installed /usr/bin/httrack then shadows your build. Run via make check, or PATH="<bld>/src:$PATH" for a manual run.
• distcheck is a required context and builds from the dist tarball, which has no .github/. A test reading a file from there must skip when it is missing, not fail. Fail only when the directory is present and the file is not, so a real deletion is still caught. Tests 226, 278, 399, 432 and 437 carry the pattern.
• Give new .test scripts set -e: the older ones predate the rule, so several local-crawl.sh calls with no set -e report PASS on any non-last failure.
• Each test runs under a 600s wall-clock guard that reports a wedge as 124. A test whose own work outlasts it raises the budget with a # TEST_TIMEOUT_AT_LEAST: N line, at column 0 within its first 40 lines, and paces itself with skip_if_out_of_budget so a host too slow to finish skips instead. The value only ever raises the budget: nothing can disarm the guard.
• Run teardown with errexit off: trap 'set +e; cleanup' EXIT. Under set -e a failing cleanup command becomes the test's exit status (#773). Keep the other signals on their own trap line, or errexit stays off for the rest of the run. The guard also resets $?, so save it first if teardown reads it.
• Never pipe into grep -q: it exits on the first match, so whatever the producer had left to write takes SIGPIPE, and under pipefail that becomes the pipeline's status. cmd | grep -q M && fail then never fires and a probe that proved nothing reads as "marker absent"; cmd | grep -q M || fail fails a test whose marker was present. bash issues one write() per line, so any match that is not on the last line is exposed. Capture the reply, assert the status line it must carry (an empty, truncated or redirected one is marker-free too), then match with a here-string: grep -q M <<<"$reply". head -c N and head -n N end the same way, and there the reds are platform-specific: GNU tail | head -c 3 survives, the uutils coreutils leg turns the same line into exit 141 with no output at all. Let the reader seek instead: od -An -c -j <skip> -N <len> file.
• Never assert a fault by signal number: SIGBUS is 7 on x86, arm, powerpc and s390x but 10 on hppa, alpha, mips and sparc. Map it with kill -l "$n".
• A fixture that needs a host to stay silent must settle (500ms), re-probe every socket, and SKIP when one answered: the powerpc and ppc64 buildds refuse TEST-NET-1 milliseconds after connect() where runners drop it. tools/hostile-net.sh runs a command on such a network.

Hard invariants

• Generated autotools files are NOT in git. configure, every Makefile.in, config.h.in, ltmain.sh, config.guess/sub, and the aux scripts are build products: .gitignored, regenerated by ./bootstrap, and shipped only in make dist tarballs (so tarball users still need no autotools). Never commit them. After editing configure.ac, any Makefile.am, or m4/, just commit those sources — re-run ./bootstrap locally to rebuild and test, but do not stage the regenerated output.
• Format only changed lines with git clang-format (clang-format 19). Never reformat untouched code: the engine was formatted by an old tool and won't round-trip.
• Byte-safe edits. src/htsconcat.c carries raw ISO-8859-1 high bytes (French comments) and the fuzz/corpus/* vectors are binary: edit those byte-wise (perl -0pi, sed), not through a tool that re-encodes to UTF-8 and corrupts them. The rest of the tree, including lang/*.txt and html/contact.html, is UTF-8 and safe to edit normally.
• Never add a matrix axis to windows-build.yml. The libhttrack job has no name:, so GitHub builds each status context from the job id plus the matrix values. That yields libhttrack (x64, Release) and libhttrack (Win32, Release), and the Protect master ruleset requires both by name. A third axis renames them, so both stop reporting on every open PR in the repo, not only the one that added the axis. Give a new Windows leg its own job with a pinned name:, or add a step to the existing job. 437_ci-windows-contexts.test fails on any such rename.

Security (HTTrack parses hostile input off the network)

• Bounds-check every copy. Overflow-safe form: put the untrusted value alone, untrusted < limit - controlled — never controlled + untrusted < limit, which can wrap and pass.
• Abort or clip is a decision, not a default. The *_safe_ helpers (strcpybuff, strlcpybuff, strcatbuff) abort on overflow. Right for our own data, wrong for anything read back from a cache, a header or the wire, where it trades a memory smash for a crash on malformed input. Clip with dst[0] = '\0'; strlncatbuff(dst, src, size, size - 1).
• A warning class is not the unsafe set. -Wformat-truncation fires only on a bounded snprintf whose return is discarded, so an unbounded sprintf into the same buffer never appears on it. Before scoping a hardening pass off compiler output, grep the unguarded forms yourself (\bsprintf\s*\(, \bstrcpy\s*\(, \bstrcat\s*\().

Changing the parser

src/htsparse.c rewrites every page of every mirror, so a regression there has the blast radius of the whole product and none of the visibility. The suite crawls fixtures the parser already handles. The damage lands in a browser, on a real site, and reaches us months later as a forum post. Treat a change here with more care than its diff size suggests.

• Widening what the parser SCANS is as dangerous as changing what it REWRITES. #1377 touched only the list of scanned mime types, and thereby carried a years-old rewrite bug into every ordinary script. #497 widened mid-tag attribute detection, which then rewrote data-* values that were never links.
• Locate your guard in the pipeline before you write it. By the time a string reaches your test it may already have been entity-decoded and truncated at # and ?. Read what happened to it upstream, rather than assuming it still holds what the page held.
• Prove it by differential against the previous release binary. Build both, crawl one fixture, diff the mirrored output. A source read that has not been confronted with two running binaries is a hypothesis.
• Pair every probe with a control that fires, including one in the narrowing direction. A mutant that wrongly rejects tells you the differential can see a loss and not only a gain.
• Name the class you mean to change, then prove the class you changed equals it.

C conventions

• Use the *t allocator wrappers, never raw libc (htssafe.h): malloct/calloct/realloct/freet/strdupt, in test and selftest code too. freet NULLs its (lvalue) argument and tolerates NULL; calloct(n, sz) keeps calloc's arg order. Only exception: storing or calling a libc symbol itself (e.g. a resolver-backend function pointer).
• Exported API is HTSEXT_API. Everything else is hidden by -fvisibility=hidden and free to change (check with nm -D --defined-only libhttrack.so). Touching an installed-header struct (see DevIncludes_DATA in src/Makefile.am) or an exported signature is an ABI break — flag and discuss, bump the soname, and prefer keeping the old entry point beside a new one.
• Windows ABI is free to break, POSIX is not. The Windows DLL ships next to the exe with no soname contract, so a _WIN32-only ABI change needs no deprecation dance; POSIX/ELF keeps the flag-discuss-bump rules.

Code & prose

• Be terse. Comment the why, in English; translate French comments you touch.
• Strip AI tells from prose (em-dash overuse, rule-of-three, filler, vague attributions). Ref: Wikipedia "Signs of AI writing". Claude Code: /humanizer.
• Behavior change → add a test. Fast path: a hidden httrack -#test=NAME engine self-test (registry in htsselftest.c; -#test lists them) driven by a tests/NN_*.test, over a slow crawl.
• A list of self-test assertions goes through selftest_queue + selftest_run_queued, which run the file's cases in one engine instead of one per assertion. Queued args reach the handler verbatim, so a case exercising an argv rewrite (CR/LF/TAB to a space, (none), quote stripping, alias expansion), or one whose output later shell logic reads, keeps assert_selftest.

Review your change adversarially (strongly suggested)

Before pushing, and when reviewing others, don't skim for bugs:

• One invariant at a time. Name a property the diff must preserve (bounds hold, cache/wire format unchanged, no use-after-free, ABI stable), then construct inputs that would break it. "General correctness" is not a charter.
• Audit tests against the spec, not the code. For each new test ask: "what buggy path would still pass this?" If you can build one, the test is confirmation-biased: assertions copied from observed output lock bugs in.
• A green suite is not evidence the old behavior was chosen. A test can be written from the same mental model as the code and pin the defect as intended: htsdns_selftest.c asserted the never-re-resolve bug #1392 fixes, and a purge guard firing on any dead link passed all 372 tests. When a fix makes you invert an existing assertion, that inversion is the finding.
• A sweep's count is a product of its corpus, so floor it in the .test. Use the real count, not a round one, and prove it by deleting a case.
• Risk areas need runtime probes. Touching hostile-input parsing, struct layout/ABI, cache/wire format, or a security path? A static or unit check isn't enough; exercise the wrong behavior at runtime. Claude Code: /review-recipe.
• Poison a canary, never compare it against zero. Checking that a neighbouring field is still '\0' cannot see the stray NUL an off-by-one terminator writes — the exact bug the canary is there for. Fill it with a non-zero byte, and prove it by killing both the stray-'X' and the stray-NUL mutant. Neither ASan nor _FORTIFY_SOURCE sees an overflow that lands inside the same struct.
• Overshoot every destination, not one. A bounds test that oversizes a single field cannot tell a per-field bound from a one-size-fits-all one, nor from a fix that bounds that field and leaves its neighbours raw. Exercise each destination the path touches, spanning at least two capacities, and check what the code actually emits before writing the expected values.

Commits

• Sign-off is mandatory. Every commit carries a Signed-off-by trailer: git commit -s (DCO, CI-enforced — unsigned commits are rejected).
• Co-Authored-By is mandatory for AI-assisted commits. Carry a Co-Authored-By: trailer naming the assistant. Attribute there, never in a PR-body footer.
• PRs are squash-merged: one commit per PR lands on master, built from the PR title and description, so those are what the history keeps. The branch's intermediate commits are not preserved.

PR descriptions

• Plain concise prose; lead with what changed and why. No What/Why/How template.
• Title says what the change does, in plain words anyone can follow without opening the diff. Not the problem behind it, and not the implementation.
• Don't restate the diff — give what it can't show: motivation, context, tradeoffs, risk.
• Write two sentences: what it does, and what a reviewer would miss. A third needs a reason.
• Verify claims against the code before you write them; flag drift, don't repeat it.
• Don't hard-wrap (GitHub reflows). No "Generated with Claude" footer. Run the prose through /humanizer.

Toolchain

C · clang-format-19 · autoreconf · shfmt + shellcheck (shell) · black + flake8 (Python)