Instruction file imported from arcaerdogar/web-crawler--multiagent-workflow (.cursor/rules/testing.mdc). Copyright stays with the author.

Testing Standards

Framework

• vitest for everything (unit, integration, API).
• supertest for HTTP/API tests against the Express app.
• No mocking of better-sqlite3 — use real in-memory or temp-file DBs. Mocking the DB hides bugs that only surface against the actual SQL engine.
• No mocking of node:worker_threads for worker-pool tests — spawn real workers.

Locations

backend/
  vitest.config.ts          ← vitest configuration (root-relative paths)
  src/__tests__/
    unit/
      normalizeUrl.test.ts
      parser.test.ts
      rateLimiter.test.ts
      validation.test.ts
      workerPool.test.ts
    db/
      search.test.ts
      jobResume.test.ts
    integration/
      crawler.test.ts
      resume.test.ts
      cooldown.test.ts
      atomicTransaction.test.ts
    api/
      startCrawl.test.ts
      jobs.test.ts
      stop.test.ts
      delete.test.ts
      restart.test.ts
      urls.test.ts
      search.test.ts
      sse.test.ts
    helpers/
      mockHttpServer.ts     ← reusable mock HTTP server fixture
      tempDb.ts             ← per-test isolated DB factory
      testApp.ts            ← builds the Express app w/ injected sseEmit for tests

DB Isolation Per Test

The db singleton in backend/src/db/connection.ts reads CRAWLER_DB_PATH env var. Tests MUST set this before the first import of any module that imports db.

Two acceptable patterns:

Pattern A — Per-file in-memory DB (fast, fully isolated):

process.env.CRAWLER_DB_PATH = ':memory:';
// Then import { db } and modules under test

Pattern B — Per-test temp file DB (when sub-modules cache db import):

import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const dir = mkdtempSync(join(tmpdir(), 'crawler-test-'));
process.env.CRAWLER_DB_PATH = join(dir, 'test.db');

Use the tempDb.ts helper to standardize this and to provide a reset between tests via db.exec(DROP TABLE ... ; CREATE TABLE ...).

The testing.mdc rule and qa-agent skill DO NOT permit altering the db singleton's lifecycle (no db.close() then re-new in production code). Tests handle this via process-level env vars + module isolation (vi.resetModules() if needed).

Mock HTTP Server

For crawler integration tests, use helpers/mockHttpServer.ts — a node:http server that serves a small set of known HTML pages with controllable links and word content. It MUST:

• Listen on 127.0.0.1:0 (random port; read from server.address()).
• Provide a clean start() and stop() lifecycle for beforeAll / afterAll.
• Allow per-route content (e.g. server.route('/page1', { html: '...', contentType: 'text/html' })).
• Optionally simulate redirects, slow responses, non-HTML content-types, and errors (for edge cases).

NEVER hit real external URLs in tests — they're flaky and slow.

Express App Test-Friendliness

backend/src/server.ts MUST split the app construction from the listener startup:

// server.ts
export function createApp(deps: { sseEmit: SSEEmit }): Express { ... }

// Only when run directly (not when imported by tests):
if (import.meta.url === `file://${process.argv[1]}`) {
  const app = createApp(...);
  const server = app.listen(3001, ...);
  installSignalHandlers(server);
}

This lets tests import { createApp } from '../server.js' and use supertest(app) without binding a port.

Edge Cases Every Test Suite MUST Cover

normalizeUrl

• All 3 scopes (hostname, registrableDomain, unrestricted) × valid input
• Invalid URLs (malformed, empty, whitespace)
• Non-http(s) protocols (mailto:, javascript:, ftp:, data:) → null
• Relative URLs resolved against base
• Base URL with port, with userinfo, with path
• Hash fragment removal
• Trailing slash (root vs non-root)
• Tracking params: utm_source/medium/campaign/term/content, ref, fbclid, gclid (all 8)
• Mixed case hostname → lowercased
• Subdomain matching for hostname scope (must NOT match)
• Subdomain matching for registrableDomain scope (must match)
• Unicode hostnames (IDN) — at minimum: don't crash
• Very long URLs (>2000 chars)
• Query params preserved when not tracking

tokenize / STOP_WORDS

• Empty string, whitespace only
• Single word, multiple words
• Mixed case → lowercased
• Punctuation stripping
• Unicode characters → split correctly
• All STOP_WORDS filtered
• Words shorter than 2 chars filtered
• Numbers preserved (alphanumeric)

RateLimiter

• rps=1: ~1000ms between calls
• rps=20: ~50ms between calls
• First wait() is immediate (no delay)
• Concurrent waits serialize correctly

WorkerPool

• Single dispatch returns expected output
• Multiple concurrent dispatches utilize multiple workers (verify by timing OR by checking each worker is used at least once)
• terminate() rejects pending tasks OR awaits them (whichever the contract says — verify behavior matches worker-lifecycle.mdc)
• Crash recovery: kill a worker, dispatch new task, verify pool respawns

SearchEngine

• Empty tokens → empty response, no SQL run (verify via spy or by deleting table)
• Single token, multi-token (with score sum)
• jobId filter
• is_active=0 jobs excluded (insert deleted job's word_index, query, verify excluded)
• Pagination (limit/offset)
• Ordering by score DESC

jobResume

• markOrphanedRunningJobsInterrupted: marks only running+is_active=1 rows; leaves interrupted, completed, deleted alone
• restoreJob: NotFound (missing jobId), NotFound (is_active=0), NoRemainingUrls (queue empty), happy path returns engine with restored state

Resume integration

• Start crawl → wait for partial progress → engine.stop() → DB has url_queue rows + visited_urls rows → restoreJob → engine resumes with same queue + visited

Cooldown integration

• Crawl URL X in job A → in job B's run, link to X is NOT enqueued, stats.skippedRecent increments
• After CRAWLER_DB_PATH allows (test can backdate crawled_at to > 30min ago), URL X IS enqueued

Atomic transaction

• Trigger an exception inside the transaction (e.g. corrupt one of the prepared statements with a wrong arg count) → verify NO rows written for that URL, in-memory state unchanged
• Successful path → verify all 4 effects (visited insert, words insert, queue delete, new queue inserts) committed atomically

API tests (each endpoint)

• Happy path with full body
• Missing required field → 400 with Zod issue messages
• Type coercion edge: limit="abc" → 400; limit="20" → 200 (coerced)
• Out-of-range numeric (maxDepth=0, maxDepth=11) → 400
• Unknown jobId → 404
• restartJob conflict cases: completed → 409, already running → 409, no remaining URLs → 409
• deleteJob soft-delete persists is_active=0; subsequent listJobs excludes
• search with jobId=deleted-job → empty results

SSE tests

• Connect, receive at least one stats event, disconnect cleanly
• Disconnect cleans up sseClients map (verify by checking the Map size or by emitting after disconnect and confirming no error)
• Heartbeat: server still writes ': ping' comment after 20s (can be tested with a shorter heartbeat interval injected for tests)

Test File Naming

• *.test.ts — vitest auto-discovers
• One concern per file
• Test descriptions in present tense: it('returns 400 when url is missing', ...)

What Tests MUST NOT Do

• Hit real external URLs.
• Write to backend/data/crawler.db.
• Leave background timers/workers/servers running (always cleanup in afterEach / afterAll).
• Depend on test execution order.
• Use Math.random() for assertions — use fixed seeds or fixtures.
• Sleep longer than necessary; for timing tests use small intervals (e.g. rps=10 means 100ms, not 1000ms).

Coverage Expectations

• Every exported function in backend/src/ has at least one direct test.
• Every API endpoint has at least 3 tests (happy + 1 validation failure + 1 edge case).
• Resume / cooldown / atomic transaction each have at least one full integration test.
• No uncovered crash-correctness path in crawler.ts or jobResume.ts.