CompanyBrain

Imported from AryanPatil5/CompanyBrain (AGENTS.md). Install upstream with npx skills add AryanPatil5/CompanyBrain. Copyright stays with the author.

AGENTS.md

Company Brain: AI knowledge engine (SOP extraction, governance, FastMCP skills). Three deployable services in one repo, no root package.json — run commands per directory.

Layout

• server/ — Express REST API (port 5001) + FastMCP (port 8080) + BullMQ/Temporal workers + crawlers. ESM ("type": "module", NodeNext); all relative imports must end in .js.
• client/ — TanStack Start (SSR) + React Router + Vite + Tailwind v4. Port 3000 (8080 is reserved for FastMCP). Uses @/* path alias to src/.
• demo-oauth-proxy/ — separate deployed service holding real Slack/Google OAuth secrets; the server never sees them. Configured via scripts/setup-demo-proxy.mjs.
• server/supabase/*.sql — migration files in filename order, applied by npm run migrate (runner: server/src/db/migrator.ts, ADR-T1). The runner owns the schema_migrations ledger and can also be run manually in the Supabase SQL Editor. Supabase is the primary DB + auth.
• Migrations are runner-owned and additive: each schema change is a NEW numbered file after the highest applied; the runner refuses duplicate version numbers and checksum-differing re-applies.
• deploy/helm/company-brain/ — Kubernetes Helm charts. docker-compose.yml runs local Postgres + Redis. deploy/docs/runbooks/ — operations runbooks (migrations, DLQ replay, re-upload recovery).

Commands

# Server (needs server/.env + docker compose up -d for Redis/Postgres)
# Boot is PROCESSES-driven (server/src/bootstrap.ts): PROCESSES unset in dev
# boots ALL processes in one process; set PROCESSES=api,mcp for isolation.
npm run dev --prefix server          # tsx watch src/bootstrap.ts (all processes)
npm run dev:api --prefix server      # tsx watch src/bootstrap.ts with PROCESSES=api
npm run dev:mcp --prefix server      # ... PROCESSES=mcp
npm run dev:crawler --prefix server  # ... PROCESSES=crawler
npm run dev:ingestion-worker --prefix server   # ... PROCESSES=ingestion-worker
npm run dev:github-sync-worker --prefix server # ... PROCESSES=github-sync-worker
npm run dev:temporal-worker --prefix server    # ... PROCESSES=temporal-worker
npm run dev:claims-backfill-worker --prefix server # ... PROCESSES=claims-backfill-worker (claims re-derivation sweeps, Phase 3 backfill)
npm run build --prefix server        # tsc typecheck + emit

# Client
npm run dev --prefix client
npm run lint --prefix client         # eslint (client; server has its own `npm run lint` — see server/package.json)
npm run build --prefix client

# CI (Phase 1 Task 4)
# .github/workflows/ci.yml gates every push/PR: lint, typecheck, builds (+ reproducibility),
# hermetic `npm test`, migration verification (pgvector service), helm lint/template,
# gitleaks secret scan, npm audit gate (scripts/ci-audit.mjs). See CI_TESTING_REPORT.md.

# Tests: custom console runners via tsx, NO test framework. Self-execute when
# run directly (pattern: `if (import.meta.url === file://${process.argv[1]})`).
npm test --prefix server             # hermetic CI run: tsx test/run-all.ts (71 suites, see test/HERMETIC_TESTING_REPORT.md)
npm run test:e2e --prefix server     # end-to-end suite
npx tsx server/test/<path>.test.ts   # any single suite (all under server/test/**)

Gotchas

• npm test (run-all) is hermetic: no live Redis/Postgres/Supabase/LLM/network — test/harness/ stubs ioredis (in-memory), global.fetch (deterministic Ollama router + loopback passthrough) and supabase client (fakeSupabase.ts). Per-suite hard timeouts; exits nonzero on any failure. Suites importing app modules that read env must import the harness first (import { installHarness } from './harness/index.js' before app imports). Some suites (e.g. infra/health.test.ts) only pass through loopback passthrough; suites that touch real infra still need it (see test/HERMETIC_TESTING_REPORT.md).
• HARMLESS-LOOKING HANG GOTCHA: never statically import app modules that transitively pull src/queue/ingestionQueue.ts (e.g. routes/documents.ts, workers/ingestionWorker.ts) in a suite that runs standalone (npx tsx test/x.test.ts). The queue module constructs BullMQ queues at module-eval, which starts a real ioredis connect; if installHarness()'s sendCommand stub lands mid-handshake, the connect never completes and every queue.add() hangs forever with zero output. Defer such imports with await import() AFTER await installHarness() (pattern in test/routes/documents.test.ts, test/workers/documentJob.test.ts). Under run-all.ts the harness is installed before any suite module is imported, so static imports are safe there — the hazard is standalone runs only.
• client/src/routeTree.gen.ts is auto-generated by TanStack Start — never hand-edit; add routes as files under client/src/routes/.
• server/src/bootstrap.ts dispatches by PROCESSES to per-process entrypoints (api, mcp, crawler, ingestion-worker, github-sync-worker, temporal-worker, claims-backfill-worker); server/src/index.ts is the API process module only (startApiServer). Production requires an explicit PROCESSES list (single-process npm start removed).
• Webhook durability (Phase 2 Task 1): webhook routes (/api/ingestion/webhook*) persist deliveries to raw_source_events (migration 035, dedupe key = sha256 of workspace+provider+external_id+event timestamp) and answer 202 {event_id, status}; the webhook-ingestion BullMQ queue is consumed by webhookEventWorker inside the ingestion-worker process (processWebhookEventJob in src/ingestion/webhookPipeline.ts, exactly-once via event status + the Phase 1 idempotency ledger). Event status is polled via GET /api/ingestion/events/:event_id. Do not call processThreadCore synchronously from routes — it lives in src/services/ingestion/webhookService.ts and runs in the worker only.
• Thread ingestion tail (Phase 3, ADR-T15): ALL thread-based ingestion funnels through processThreadTail in src/ingestion/documentPipeline.ts — used by processThreadCore (durable webhooks) AND by all six legacy crawlers (src/services/crawlers/{slack,github,linear,zendesk,email,database}.ts). The tail persists source document + chunks → grounded knowledge_claims + claim_evidence (char offsets) → schema-validated SOP extraction. Each crawler keeps its own skills_sops insert shell (per-provider defaults, + confidence_score from migration 037) and then links claims via linkSopClaimsBestEffort (warn-only; the webhook path uses the strict throwing linkDocumentClaimsToSop so the event ledger retries). Every crawled/webhook thread therefore produces documents/chunks/claims exactly like the upload pipeline, and GET /api/sops/:id/claims returns real data. Entity/relationship confidence in entityResolver.ts is DERIVED from sighting volume (migration 038: times_seen), never hardcoded 1.0.
• Upload pipeline (Phase 3): POST /api/documents/upload → content-addressed storage → document-ingestion queue → processDocumentIngestionJob (src/workers/ingestionWorker.ts), which checkpoints extraction_stage through parsing → chunking → embedding → claims → completed (real stage markers, migration 036 enum). Duplicate re-uploads return 202 + deduplicated: true with the existing document_id (23505 caught in src/routes/documents.ts) — never a 500; re-upload is the recovery path for failed rows. Scanned/empty PDFs land in the explicit ocr_required terminal stage; text is never fabricated. Similarity in searchVectorContextDLAC is honest (?? null), never a fake 0.9 fallback.
• Claims backfill worker (Phase 3, ADR-T15): claims-backfill-worker process owns the claims-backfill BullMQ queue (one batch job per sweep via a BullMQ v6 job scheduler, CLAIMS_BACKFILL_INTERVAL_MS default 60s, health port 5007 CLAIMS_BACKFILL_WORKER_HEALTH_PORT). Core logic in src/ingestion/claimsBackfill.ts: scans source_documents where extraction_stage='completed' AND claims_derived_at IS NULL AND claims_backfill_failures < 3 (migration 039), re-derives knowledge_claims/claim_evidence from stored document_chunks via the same idempotent store the pipeline uses, then stamps claims_derived_at/claims_derived_version. Progress lives in the DB checkpoint, not the queue — crashed/restarted sweeps re-pick the same rows and the (workspace_id, source_document_id, chunk_id, claim_text_hash) unique key makes re-derivation a no-op; poisoned docs are quarantined after 3 failures. Both live ingestion paths (processThreadTail and processDocumentIngestionJob) stamp claims_derived_at after successful claim extraction so only genuinely-missing docs become candidates.
• Embedding backfill (Phase 4 T2): the embedding-backfill BullMQ queue (job scheduler every EMBEDDING_BACKFILL_INTERVAL_MS default 60s) runs INSIDE the ingestion-worker process (no topology change). Core logic in src/ingestion/embeddingBackfill.ts (queue-free, hermetically tested in test/workers/embeddingBackfill.test.ts; the processor seam processEmbeddingBackfillJob lives in the core, the worker module src/workers/embeddingBackfillWorker.ts is the thin queue citizen). Each sweep is ONE bounded keyset page (id DESC, WHERE id < cursor, EMBEDDING_BACKFILL_BATCH_SIZE default 100, cap 500) that re-embeds only stale chunks: embedding_model/embedding_version ≠ current provider's (from getEmbeddingProvider(), never env), missing/malformed vector, or content_hash ≠ canonical hashContent(content) (force bypasses). Safety invariants: the ONLY write is an atomic conditional UPDATE (WHERE id + workspace + content + content_hash as observed) so concurrent ingestion writes are never overwritten (concurrent_modifications reported); current chunks make zero provider calls; per-chunk failures are isolated and counted honestly (retryable → retried next sweep; non-retryable → in-process bounded quarantine cleared at worker start); cursor is an optimization only — resume persists via scheduler-template re-upsert, a stale cursor just re-scans and skips; cost metering is best-effort recordUsage with honest zeros (never fabricated). No migration needed (columns from 027 + 036).
• Health endpoints (Phase 0 Task 2, server/src/services/health.ts): every process serves GET /health with structured JSON — {status, process, version, uptime, pid, startedAt, dependencies} — always HTTP 200 while the process is alive; dependencies fail individually (ok/unavailable), never crash, never expose secrets. Ports (env-overridable): API 5001, crawler 5002 (CRAWLER_PORT), MCP 5003, ingestion-worker 5004, github-sync-worker 5005, temporal-worker 5006, claims-backfill-worker 5007 (CLAIMS_BACKFILL_WORKER_HEALTH_PORT).
• Observability (Phase 0 Task 9): server/src/logger.ts is the only logging path (no console.* in runtime code) — structured JSON to stdout with {timestamp, level, service, process, correlationId, pid}; secrets redacted by key name and value pattern (Bearer/API keys/passwords/cookies/OAuth tokens/provider keys/Slack/GitHub tokens/JWTs/PEMs/URL creds); LOG_LEVEL filters. Correlation IDs flow via correlationIdMiddleware (preserves x-correlation-id/x-request-id, echoes response header) through AsyncLocalStorage (runWithCorrelationId). server/src/config/otel.ts starts the OTel SDK only when OTEL_ENABLED=true (OTLP HTTP exporter to OTEL_EXPORTER_OTLP_ENDPOINT); otherwise no-op; shutdownOpenTelemetry() runs on graceful shutdown. Metrics/Prometheus remain Phase 9.
• Cost meter (Phase 0 Task 10, server/src/services/costMeter.ts): every successful LLM request records provider/model/token counts/estimated cost/latency/workspaceId/correlationId into usage_meters (needs migration 031_usage_meters_detail.sql applied — on cloud Supabase it must be run via the SQL Editor; until then persistence degrades to the minimal row and logs). recordUsage NEVER throws on persistence failure (best-effort, 2s timeout). checkQuota is report-only via COST_QUOTA_<WORKSPACE_ID_UPPER_WITH_NON_ALPHANUMERICS_AS_UNDERSCORES> env (unset = no quota); it never blocks. Billing/budgets are Phase 9. Hermetic tests live in server/test/infra/costMeter.test.ts and swap in an in-memory store via setUsageStoreForTest (no live infra).
• Env: server/.env (see .env.example) needs SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY, VAULT_SECRET_KEY (openssl rand -hex 32), OPENROUTER_API_KEY. Prod refuses the zero-workspace seed ID (DEV_SEED_WORKSPACE_ID).
• Auth relies on Supabase Custom Access Token Hook (public.custom_access_token_hook, migration 015) — must be enabled in dashboard or API auth breaks.
• Migrations are runner-owned and additive: each schema change is a NEW numbered file after the highest applied; the runner refuses duplicate version numbers and checksum-differing re-applies.
• COMPANY_BRAIN_CRITICAL_REVIEW.md (root) is a candid architectural audit of known limitations (e.g., OpenAPI skills return compiled_skill_dispatched without real execution). Read it before claiming features work as-advertised.
• OAuth demo mode: scripts/setup-demo-proxy.mjs writes demo-oauth-proxy/.env + updates server/.env (DEMO_PROXY_* vars). Both .envs are gitignored.