ZuiGo-Engine

Imported from ZuiGO/ZuiGo-Engine (AGENTS.md). Install upstream with npx skills add ZuiGO/ZuiGo-Engine. Copyright stays with the author.

RankEngine — Agent Context & Session State

This file is the durable memory for AI coding sessions. READ THIS FIRST at the start of every session. Update it at the end of every session with what changed and what's left.

Current state & guardrails (verified 2026-10-14 — trust this over the log below)

The long "Status" log and "Next up" list further down are HISTORICAL (last real entry 2026-08-13). They mention things that no longer exist, so check any item against the code before acting on it. Do not "resume" work from them blindly.

Removed on purpose (do not recreate): the sandbox connector layer (services/connectors/*, git_static_connector, wordpress/replica connectors), staging/replica flows, the old agent-swarm framework, deploy/demo/ (merged into the root docker-compose.yml), and POST /api/actions/{job}/apply-sandbox (now 410). Approved changes ship as a GitHub PR (create_github_pr, needs GITHUB_TOKEN) via /api/actions/{job}/apply or /api/sandbox/suggestions/{id}/apply.

Agents: services/agent_runtime.py is the loop (budget, checkpoints, loop guards); agent_planner.decide() picks the next tool; agent_tools/ holds the registry (ToolSpec objects — use attributes, not spec["..."]); agent_swarm/ = domain agents that are AgentRuntime + role prompt + tool allow-list. Runs are executed by build_agent_for_run() (API and arq worker).

Rules for AI sessions (each one exists because it went wrong):

1. Run .venv/bin/python -m pytest backend/tests -q before every commit; it must be fully green. test_import_integrity.py fails on any dangling import, including lazy imports inside functions — never delete or rename a module/ function without git grep-ing every reference first.
2. Pipeline stages swallow exceptions (_stage(..., fallback=...)), so a broken stage looks like "no data". After touching a stage, run it, don't just import it.
3. Most tests run with no GROQ_API_KEY, i.e. the deterministic fallback path. Anything on the LLM path (planner, groq_client, analysis stages) needs a test that patches a key and a fake client.
4. Do not swap a parser/library for regexes "for performance" (this silently dropped <picture>, <video>, <object> and width/height from the content classifier). If you must, keep the existing tests passing.
5. Never commit .env*, *.log, seed/, or keys. Do not log request URLs or query strings (API keys travel there); logging_setup redacts as a backstop.
6. Keep this file honest: when you delete or add a subsystem, edit the section above in the same commit.
7. Results must come from measurements. Never default a missing measurement to a "good" value (e.g. score = 100, .get("score", 100)): use None, exclude it from roll-ups and show "not measured". No hand-written confidence numbers; findings carry basis (measured | rule | heuristic) via seo_rules.basis_for.
8. Judgement thresholds (title length, thin-content words, click depth, ...) live in services/seo_rules.py and every finding states the numbers it used in evidence. Do not scatter new magic numbers through services.
9. Accuracy is tested against ground truth: backend/tests/golden/ crawls a local site whose true facts are known. When you find an accuracy bug, add the failing case to golden/site.py and an assertion first, then fix the code.
10. Every audit score is now a measured rate (unweighted mean of measured components) or None. Scores that only apply to some sites carry applicable (local_seo, programmatic_seo, international_seo) - "not applicable" is not a failing grade. test_no_fabricated_results.py fails if a missing score defaults to 100 or an invented "after" value reappears.

Project

SEO analysis engine: crawl a site, classify content, analyze, recommend, approve changes, generate dummy site + reports, chat. Python 3.14 FastAPI + MongoDB + Redis + Chroma + Arq worker + Playwright. Repo: /Users/macbook/RankEngine-AI-Simple (git, remote https://github.com/ZuiGO/rankengine-seo.git).

Status — HISTORICAL LOG (last update: Phase 5 Sandbox Comparison View 2026-08-13; git_static_connector tests fixed; node --check clean; pushed 22f1753)

• AGENT RUNTIME ROUND (pushed as one commit, see git log; services restarted via launchd):

  • Autonomous SEO agent: POST /api/agent/runs (backend/routes/agent.py, models/agent_schemas.py) with goal/scope/urls/budget_credits/max_steps/checkpoint_policy; arq task agent_run (worker.py) runs AgentRuntime.start(run_id) (services/agent_runtime.py) — loop: build state -> planner decision (services/agent_planner.py, Groq) -> tool execution (services/agent_tools.py: crawl_urls, run_analyzers, generate_suggestions, apply_to_sandbox, verify_changes, read_current_state) -> fact extraction (agent_memory.py: agent_runs/agent_episodes/agent_facts collections, indexes in db/mongo.py). Approval gate: apply/verify steps pause as waiting_approval with pending_payload; POST /api/agent/runs/{id}/approve resumes (checkpoint_policy=every_apply). Loop guard (_decide_with_guard) blocks repeated identical calls unless the facts fingerprint changed.
  • single_page_service.py + POST /api/sandbox/single-page + GET status: one-page crawl -> Groq suggestions -> git_static_connector apply to the static-replica sandbox -> snapshot comparison; frontend "Single Page Analysis (Demo)" button on Analyze (placeholder/chips stay privacy-safe yoursite.com/example.com) -> sandbox-comparison tab auto-populated (base64 screenshots, score delta, field table, history).
  • snapshot_service.py: capture_snapshot runs vercel curl with cwd=sandbox/static-replica, injects <base href> + inlines CSS/images (base64) so snapshots render without Vercel SSO.
  • git_static_connector.py: h1 + alt_text field support (hero regexes), preview_url now appends /products/railways; test updated.
  • config.py: agent_max_steps=15, agent_default_budget=100.0, agent_checkpoint_threshold=50.0.
  • Sandbox hardening: sandbox/wp-docker/ docker-compose now internal-only (8090 not published) behind nginx-basicauth on 8091, one-shot wpcli provision.sh (Yoast SEO), robots.txt mount, README + verify_rest_api.sh; sandbox/static-replica gitlink bumped (769a44e, still no .gitmodules mapping).
  • Stability fixes (same round): verify_changes result truncation fixed (strip screenshots/raw_history, filter field_comparison to changed/meaningful, cap 12 x 80 chars; MAX_RESULT_CHARS 4000 -> 6000 in agent_tools.py + agent_runtime.py) — live-verified seo_score {old:50,new:65,delta:15} + facts seo_score_delta; full loop crawl -> analyzers -> suggest -> apply (approval) -> verify -> complete.
  • Full test suite 409/409 passing; node --check clean.

• PHASE 5 — PAGE-LEVEL COMPARISON VIEW:

  • backend/services/snapshots/comparison_view.py (NEW): get_comparison_data() fetches baseline (Phase 2) + latest (Phase 4) snapshots from sandbox_snapshots, builds field-by-field change table (applied → Changed/Unchanged based on value equality), calculates composite SEO score delta (deducts for missing title/meta/alt/schema), enriches raw history from sandbox_audit_logs with field names + Vercel preview URLs.
  • backend/routes/sandbox.py: GET /api/sandbox/comparison endpoint added.
  • frontend/index.html: #tab-sandbox-comparison with SEO Score Impact card, Field-by-Field Analysis table (baseline vs current, Changed/Unchanged colour badges), Visual Comparison side-by-side panels (Baseline Phase 2 / Current Post-Apply screenshots), Raw History Trail sidebar (✅ Applied / ↩️ Rolled Back per event with commit hash + Vercel preview links). Hidden tab button with style="display:none" shown programmatically.
  • frontend/app.js: loadSandboxComparison(), hash route #sandbox-comparison in restoreFromHash(), VALID_TABS extended.
  • backend/tests/test_git_static_connector.py: Completely rewritten with mocks for _modify_source_file, _commit_changes, _trigger_vercel_deploy, capture_snapshot — correct field type names (title, alt_text), 5-tuple unpack — all 4 pass.
  • Verified at 1440px: Title → Changed (green), Schema Markup → Unchanged (grey), Footer Copyright → Changed (green), Alt Text → Unchanged (grey); 4 field rows; 3 history entries; 0 console errors.
  • Accessible at http://localhost:8001/app#sandbox-comparison.

• USER VISIBILITY ADJUSTMENTS ROUND:

  • Hidden Crawl Graph Topology: Set display: none !important on .graph-header-bar and #crawl-graph-section in frontend/index.html. End users no longer see the 3D topology visualizer header or WebGL canvas.
  • Hidden Style Guide: Removed Style Guide tab button from the top tab strip, Quick Links grid, and RAIL_TAB_LABELS, TAB_GUIDES, and VALID_TABS navigation maps in frontend/app.js.

• PERFORMANCE & REDUCED-MOTION SAFEGUARDS ROUND:

  • Defer 3D Graph Initialization: Enforced requestIdleCallback / setTimeout for 3D force graph initialization in loadLinks(jobId) so primary stat cards, summary metrics, and data tables paint and become interactive first.
  • Universal Reduced-Motion JS Count-Ups: Updated animateCountUp to check isReducedMotionSet() and instantly render target text without frame ticking when reduced motion is preferred.
  • Comprehensive Reduced-Motion CSS Rules: Added @media (prefers-reduced-motion: reduce) block disabling transitions and keyframe animations for radial gauges, stacked health bars, longest link fills, skeleton shimmers, and row checked pulses.
  • Zero Business Logic Mutation: Preserved all backend data fetching, state management, and business logic intact.

• LINK HEALTH GAUGES, STACKED BAR & VISUAL COMPARISON ROUND:

  • Internal Ratio Radial Arc Gauge: Replaced text percentage in 4th stat card with a radial arc SVG gauge showing internal vs external link proportions.
  • Proportional Horizontal Stacked Bar: Added a stacked distribution bar above summary cards rendering OK (--status-ok), Broken (--status-broken), Redirect (--status-redirect), Blocked (--status-blocked), and Unreachable (--status-unreachable) segments for at-a-glance health shape.
  • Staggered Tabular-Mono Count-Ups: Count-ups on reveal staggered by 40ms per card across top stat cards and summary cards using JetBrains Mono tabular monospace digits (tabular-nums).
  • Longest Links Visual Bar Comparison: Transformed bullet list into horizontal progress bars scaled proportionally to the longest link's character count ((length / maxLen) * 100%).
  • Sortable Table Columns: Made Status, Code, and Length columns sortable with click handlers and direction indicators (▲ / ▼).
  • Linked From Page Jump: Clicking any Linked From source page URL cell switches to the Pages tab, filters by that page URL, and highlights it.
  • Live Relative Timestamp: Displays "Last checked" as a live relative label ("checked 4m ago") updating every 30s with native tooltip on hover for absolute timestamp.

• LINKS TABLE POLISH & ACCESSIBILITY ROUND:

  • Status Pills & Icons: Rendered status pills paired with icons for every state (✓ OK, ✕ BROKEN, ↳ REDIRECT, 🚫 BLOCKED, ⚡ UNREACHABLE, ↗ EXTERNAL, ⋯ UNCHECKED) using semantic status colors.
  • Newly Resolved Link Row Pulse: @keyframes rowCheckedPulse (.row-just-checked) gives a 1.2s soft highlight pulse when a link transitions from unchecked to a resolved status.
  • Interactive Status Filter & Summary Cards: Added #link-status-filter and made summary cards in Link Health clickable to set the filter dropdown, re-render, and scroll to matching table rows.
  • Row Density Toggle: Added #link-density-toggle-btn to switch between Compact View (padding 4px 8px) and Expanded View (padding 10px 14px).
  • Accessible Click-to-Copy & Full Tooltip: Rendered URL cells as .page-url-copyable with full un-truncated title tooltip, tabindex="0", role="button", copy icon, and keyboard event handlers (Enter/Space key copies full URL to clipboard with toast notification).
  • Skeleton Shimmer Loading Rows: Added .skeleton-shimmer-row placeholders while link health data loads.

• APP SHELL & Z-INDEX BUG FIX ROUND:

  • Fixed Links Tab Text Bleeding Bug: Applied min-width: 0; overflow-x: auto; max-width: 100%; to .results-section, #tab-links, and data tables, enforced word-break: break-all; on .page-url-cell, and set position: sticky; z-index: 10; isolation: isolate; background: var(--bg-base); on .left-rail & .right-rail with solid --bg-surface panel background tokens. Zero text bleeds through/behind right-rail panels.
  • Sidebar Mini-Stats Count-Up: #rail-pages-count, #rail-content-count, and #rail-actions-count animate count-up on load and whenever values re-evaluate on re-crawls.
  • Sliding Nav Active Indicator Bar: Added #rail-nav-indicator which smoothly translates vertically (transform: translateY(...)) and resizes height dynamically over 0.25s --ease-entrance when switching tabs instead of popping.
  • Tip of the Day Auto-Rotation: Cycles through 6 real SEO tips every 12s with a 200ms crossfade (.fade-out / .fade-in), manual prev/next controls, and pause-on-hover/focus.
  • Honest Activity Feed Slide-In: Preserved honest empty state when no events exist (zero fake data). Real incoming crawl events slide in from top with @keyframes activitySlideIn and a temporary indigo highlight flash.
  • Quick Links Grid Polish: Added 14px SVG icons to each button and active press scaling (transform: scale(0.96)).

• URL FORM & PER-ICON MICRO-ANIMATIONS ROUND:

  • Live Inline URL Validation: #url-input displays real-time check (✓ in teal-green .valid) or cross (✕ in red .invalid) indicator with accent focus glow (--accent). Auto-prefixes https:// if omitted.
  • Compact Inline Progress Button State: "Run free site audit" transitions into inline scanning spinner (Starting crawl... + radar pulse) while POST /api/analysis connects, then hands off cleanly to live crawl screen.
  • Tappable Try Chips: .example-chip buttons for example.com and books.toscrape.com with active hover states.
  • 6 Feature Card Micro-Animations: Reordered to match dashboard navigation sequence (Technical SEO → On-page SEO → Content & Performance → Links & Quality → AI & Local Visibility → Actions & Reports) with tailored SVG micro-animations on hover (clock ticks 360°, heading lines expand, trend line draws, chain links snap, AI sparkle scales up, checkmark stroke self-draws).

• DESIGN TOKENS, STYLE GUIDE & 3D CRAWL GRAPH + LANDING HERO ROUND:

  • Added design tokens for background tiers (--bg-base, --bg-surface, --bg-elevated), accent tones, 7 semantic status colors (--status-ok, --status-broken, --status-redirect, --status-blocked, --status-unreachable, --status-external, --status-unchecked), 4 lit-edge elevation levels, instrument grid texture, motion tokens, and JetBrains Mono tabular-figure monospace numeric typography.
  • Added interactive Style Guide route /app#style-guide displaying design tokens, typography comparison, elevation cards, motion speed triggers, and reduced-motion detection.
  • Implemented 3D Crawl Graph endpoint GET /api/links/{job_id}/graph and frontend component with D3 physics force-directed layout, lazy-loaded WebGL, load-in discovery animation, pausable auto-rotation, node hover highlighting, clicked node side panel drawer (#graph-node-panel), and reduced-motion/WebGL fallback.
  • Enhanced Marketing Landing Hero: #hero-radar-bg subtle low-contrast radar sweep (WCAG AA compliant), one-time decode-in focus reveal for "the truth", interactive 3D perspective cursor tilt on #sample-report-card, dual SVG radial arc gauges (79/100 healthy vs 56/100 warning), scroll-triggered progress bar fill, 3 staggered entrance tags, and ambient background canvas graph.
  • Tests: backend/tests/test_crawl_graph.py added. Full test suite 391/391 passed. node --check frontend/app.js clean. launchd server restarted.

• CLEAN-SCREENS ROUND (rails/landing, committed summary below):

  • setDashboardVisible(visible) toggles #dashboard-grid.hidden + body.no-dashboard; Analyze page, crawl screen and /app with no job are clean; results view and jobless Sites view show the dashboard. body.jobless hides #tabs + #rail-nav (jobless "Past Analyses" header = #results-* + Sites tab without the tab strip).
  • Bare-tab hash routing: restoreFromHash() handles #sites -> openPastAnalyses() (loadSites + jobless header "Past Analyses"). Landing nav + CTA "Past analyses" link to /app#sites — no real job data on the public landing.
  • Chat widget now floats bottom-right for all non-docked states (position:fixed; bottom:24px; right:24px; panel margin-bottom:10px, transform-origin:bottom right) — old top:76px overlapped the Analyze header and the landing CTA. Docked (>=1280 + job open, body.app-wide) restores margin-top:8px / transform-origin:top right.
  • Verified: clean-screens suite + 375/1024/1280/1440 viewport suite, zero console errors; crawled example.org end-to-end, jobs hard-deleted after.

• EMAIL-ROUND (backend + frontend):

  • notifications.py: _send_sync branches port 465 -> smtplib.SMTP_SSL (implicit TLS), else STARTTLS (default) or plain; NEW try_send_email(to, subject, body, attachment) -> (sent: bool, error: str|None) returning the REAL failure reason; email_report() returns (bool, error) with per-step messages (Job not found / Report rendering failed / PDF rendering failed).
  • routes/reports.py POST /{job_id}/email: 502 body {"error": <real reason>}.
  • routes/app_settings.py: SmtpTestRequest{to} + POST /api/settings/smtp/test -> {sent, error} (unconfigured -> "SMTP host not configured. Add it in Settings → Email."). Frontend Settings → Email: #smtp-test-to + #smtp-test-send
    • #smtp-test-status + #smtp-test-error (renders real reason); report-email handler unwraps data.detail.error.
  • Gmail ready: host smtp.gmail.com port 587, app password (16-char), STARTTLS; stored DB-first in app_settings key smtp (Settings tab), .env fallback.

• STOP-ANALYSIS ROUND (cooperative cancellation + purge):

  • services/job_cancel.py (NEW): JobCancelled(job_id) + check_cancelled(job_id) — reads cancelled:true flag OR status cancelled on the analysis_jobs doc; NEVER raises on DB/network errors (best-effort; tests pass a fake DB).
  • services/job_cleanup.py (NEW): hard_delete_job(job_id) — full cascade extracted from sites.py hard-delete (JOB_COLLECTIONS rows by job_id + target_job_id, child competitor analysis_jobs docs $in deleted, their rows, crawl_schedules history $pull, Chroma job_{id}, dummy_site/downloads dirs). sites.py hard-delete now calls it.
  • Checkpoints: crawler (update_progress tick, batch-loop top, after browser.close before mobile phase), link_checker (every 20 links), analysis pipeline start, _progress() / _stage() re-raise; except JobCancelled in the pipeline: marks status=cancelled + "Cancelled by user" + completed_at, purges via hard_delete_job (guarded), sends NO alert emails.
  • POST /api/analysis/{job_id}/cancel: 404 missing; 409 unless queued|running; sets flag + cancelled_at + "Cancelling...", log_audit("analysis_cancelled").
  • Frontend: #stop-analysis-btn (btn-danger "Stop") + #stop-analysis-note on the progress card; stopCurrentAnalysis() (disables, "Stopping..."); pollJob handles status==="cancelled" AND resp.status===404 (purge race) -> showCancelledCard(); card: "Analysis Cancelled", .status-cancelled badge, button flips to btn-secondary "Back to Analyze" (RE-ENABLES disabled state, onclick -> #new-analysis-btn). CSS: .btn-danger, .progress-actions, .status-cancelled.
  • Tests: backend/tests/test_cancel.py (16: check_cancelled x4, cancel endpoint x5 incl 404/409, pipeline pre-cancelled purge x1, hard_delete_job cascade incl child jobs x1, smtp x5: 465->SMTP_SSL, 587->starttls, error surfacing, unconfigured, test endpoint). FakeDb __getitem__ + _coll-namespaced store. Full suite 373 -> 389. node --check clean.

• Live-verified (2026-08-12, Playwright 1440): SMTP test UI shows real error when unconfigured; mid-crawl Stop -> "Analysis Cancelled" card -> job 404 after purge -> no trace in /api/sites -> Back to Analyze clean screen; zero console errors except the EXPECTED one 404 poll tick after purge (frontend handles it; inherent to the delete-on-cancel design). Viewports 375/1280: no overflow, stop button visible.

• OPS: launchd server/worker restarted with new code. Docker demo re-pinned to 8002 (DEMO_PORT=8002 docker compose up -d --build app from deploy/demo) — the demo compose default ${DEMO_PORT:-8001} had squatted the dev server's 8001 after a plain up; keep demo on 8002 (8001 stays the launchd dev server). Demo health ok on 8002.

• Live sites on 8001: cced02d1-8a39-41b9-a523-9489231d823b (zuigo.ai), 6294eef9-1f42-4974-babb-170165f7736f + 1c460760-4214-4597-8fc8-98a868a6be44 (fluidcontrols.com x2).

• KEYWORD RELEVANCE ROUND (committed; backend change -> server/worker restarted via launchd):

  • keyword_engine.py: canonicalize() merges near-duplicates via snowball stems ("Double Ferrule Fittings" == "double-ferrule-fittings" == "double ferrule fitting"); corpus_ngrams ranked by count x sum-of-token-IDF (sitewide nav phrases like "industrial fittings" lose to product-specific ones); _passes_intent gate rejects stopword/modifier/nav boundaries, number+unit tokens (10mm, 40bar; material grades like 316/316l survive), nav-only combos; apply_modifiers skips modifier-word cores ("india india" dies, "union manufacturer" survives); GREEDY_SLUGS extended (services/ values/vision/mission/history/leadership/press/media/events/testimonials/support); multi-word cores outrank single-word ones (_core_score = weight x 1.8 for >=2 words).
  • job_keywords collection (NEW): versioned cache (KEYWORD_CACHE_VERSION = 4) written once per analysis; get_smart_keywords(rebuild=True) in the pipeline wave2 keywords stage (use_llm=True once per NEW job). All consumers (fetch_all_insights merge, run_serp_rankings, suggested-keywords, chat, competitors) slice the SAME ranked list; cache hit is ~0.0007s vs ~2.5s build and removes repeated SE Ranking spend.
  • llm_polish: fact-anchored (candidates must share >=2 tokens with a seed or appear verbatim in corpus text), one retry on unparseable output. ROOT CAUSE FIXED: Groq gpt-oss-120b burned the whole 512-token budget on reasoning (finish: length, reasoning_tokens=510) with long seed lists -> empty content; now max_tokens=2048 + seeds capped at 40, and base list reserves slots so LLM extras aren't truncated.
  • keyword_extractor.py: weighted fields (title 3x, meta 2x, URL slug 2x, body 1x) + 2-3 gram phrase scoring with token-IDF prior; all callers keep (job_id, top_k) API.
  • seo_insights.py CACHE_VERSION 15 -> 16; sites hard-delete cascade gains job_keywords; analysis summary gains total_smart_keywords.
  • New backend/tests/test_keyword_relevance.py (19 tests: canonical merge, intent gates, idf-prior ranking, modifier-core gate, weighted extraction, cache build/hit/version/ rebuild with no-SE-spend assertion, LLM fact-anchor accept/reject/corpus-verbatim); test_suggestions.py::test_common_terms_ranked updated (phrases now outrank the bare unigram — old test asserted unigram-only behavior). Full suite 354 -> 373.
  • Live-verified on fluidcontrols 6294eef9: top-20 now real product phrases (male elbow, pipe clips, ball valves, drain valve, flow series, union cross/elbow, coupling cap, gauge valves) — corporate junk (services, fc values/vision/mission) gone; suggested- keywords returns 45 incl. 25 LLM purchase terms ("globe isolation valve price", "compact non return valve manufacturer india"); insights refresh (v16) = 22 keywords, 12 site-derived prepended, source "se-ranking+site"; Playwright SEO Insights tab zero console errors.

• TINY FIX (committed with this round): crawl screen leaked the previous audit's stats — showProgress() never reset #cs-pages ("pages crawled" kept the old job's count until the first poll tick). Now resets to "0" and also empties the rail Live-activity feed (old crawl events cleared, then the new "Analysis queued" entry pushes). Verified: node --check clean; Playwright end-to-end (2 real crawls submit+complete, second submit shows reset, jobs hard-deleted, zero console errors) + injected-stale-state check (cs-pages 261 + 2 feed items -> showProgress() -> "0" + empty feed). Frontend-only, no server restart needed.

• Bigger chatbot + fresh Docker demo (353b72b, pushed; no backend logic change except crawler.py no-sandbox):

  • Chatbot: floating panel 390 -> 520px wide, chat-box 420 -> 560px tall (also fixed a leftover .chat-panel .chat-box { height:400px } override that had silently capped it), toggle 13px/24px @15px font, messages/input 15px, chat-table 12.5px. Docked (>=1280): right rail minmax(264,288) -> minmax(304,348)px, panel min(540px, calc(100vh-190px)), box min(480px, calc(100vh-280px)). Note: with NO job open the right rail auto-places into the middle grid column (results-section display:none) and becomes ~780px wide — pre-existing behavior, deliberate. Playwright at 375/1024/1280/1440/1920: zero overflow, floating 520x560 (capped by viewport on short windows), docked 314x528/480 box, zero console errors.
  • DOCKER DEMO (deploy/demo/docker-compose.yml == THE demo stack now; the repo-root docker-compose.yml is back to dev-infra only: mongodb community-server + redis + neo4j): services mongodb (mongo:7, NO host- published ports), redis, one-shot mongo-seed (mongo:7 image, mongorestore --nsInclude='rankengine.*' --drop /seed/dump, mounts ../../seed:/seed:ro), one-shot seed-vectors (python scripts/seed_demo.py -> vector_service.index_job_vectors per completed job; idempotent wipe+ rebuild), app (port ${DEMO_PORT:-8001}:8000) and worker (python -m arq backend.worker.WorkerSettings) sharing the image (build context ../.., Dockerfile at repo root). env_file: .env.docker (example committed; real keys gitignored via .env.docker + seed/ patterns). Dockerfile: py3.12-slim + libmagic1/libglib/fonts-dejavu + playwright chromium + install-deps; crawler launches chromium with args=["--no-sandbox"] (both sites in crawler.py) so crawls work as root in containers.
  • COMPOSE GOTCHA (cost ~1h): docker compose v5 with a project-dir .env injects that .env into every container and it BEAT the anchor env_file: (real keys + localhost URIs leaked into containers, mongod unreachable). Fix: the demo compose lives in deploy/demo/ whose project dir has NO .env — isolated from the dev .env entirely. Also: mongorestore needs --nsInclude= (not --db) + the dump-root arg; docker compose run path caveats on env_file in anchors; seed script must sys.path.insert(0, repo-root) + call connect_db(settings.mongodb_uri) (get_db returns None otherwise).
  • Seed data: mongodump of local rankengine DB -> seed/dump (165MB, gitignored; excludes app_settings + gsc_credentials = no secrets). 3 jobs: zuigo.ai (24pg), fluidcontrols.com 6294eef9 (261pg, full audits)
    • a 2nd fluidcontrols 1c460760. Config via python -c script expected key=value lines; plain split("=") readers can fail on trailing/odd chars (used regex ^KEY=(.+)$).
  • Live-verified on port 8002 (8001 busy with launchd): health ok, sites list = 3, vectors 72+500+500 indexed (Gemini 429 -> hash fallback, still indexes), overview/insights(9)/competitors/report tabs render, live Groq chat answered "261 pages, grade D" with 7 clean bullets (no raw pipes/asterisks), zero console errors, idempotent second up (seed one-shots stay exited, restart: "no", app/worker unless-stopped). README has a Docker demo section. Full reset: cd deploy/demo && docker compose down -v && up -d --build.

• UI round 3 (012dd05, pushed; backend prompt change -> server restarted via launchd):

• UI round 3 (012dd05, pushed; backend prompt change -> server restarted via launchd):

  • Screen fit: left rail appears >=1024px (was >=1100) and the top tab strip is display:none >=1024px (the rail's 13-button nav — active state + badges — is the navigator; strip clicks still work via switchTab() since programmatic .click() fires handlers on hidden elements). Right rail + docked chat move to >=1280px (was >=1360); columns narrower (minmax(176px,206px) | 1fr | minmax(264px,288px)); .main padding clamp(16px,3vw,28px); rails max-height: calc(100vh - 96px); floating chat panel max-height: min(460px, calc(100vh-96px)); docked panel min(400px, calc(100vh-190px)), chat box min(320px, calc(100vh-280px)). Tab strip flex-wrap: wrap below 1024 (no more invisible sideways scroll). Playwright: zero horizontal overflow at 375/1024/1280/1366/1440/1920; rail-nav verified to switch every tab with the strip hidden (pages 24 rows loaded via rail click at 1280).
  • Chat replies: CHAT_FORMAT_RULE constant in chat_service.py appended to SYSTEM_PROMPT + GENERAL_SYSTEM_PROMPT (verdict sentence first, <=8 dash bullets, numbers inline, NEVER markdown tables/pipes/bold; full-site chat system = SYSTEM_PROMPT + FULL_SITE_PROMPT so the rule reaches it). Frontend safety net: renderBotReply() (app.js) used for bot messages — escapes first (XSS-safe), collapses stray | table blocks into <details class="chat-collapse">Full breakdown</details> with .chat-table, - /* runs -> .chat-bullets ul/ol, **bold** -> strong, URLs -> links. New tests backend/tests/test_chat_format.py (5); suite 354/354; node --check clean; live-verified injecting the real markdown reply (2 collapsible tables, 3 bullets, 1 strong, 1 link, zero raw pipes/asterisks, zero console errors).

• UI round 2 (frontend-only, no backend change, no server restart needed):

  • Full rails: .main 1120 -> 1440px; #dashboard-grid wraps results: 1-col <1100px, 2-col 232px + 1fr >=1100px, 3-col 232px + 1fr + 308px >=1360px. Left rail (sticky): job context card (renderRailJob — health ring as conic-gradient from summary.summary.score/metrics.score when it's a number, else NO ring; pages / content items / actions lines), quick nav (buildRailNav mirrors all 13 tabs, active state + per-tab count badges via setTabBadge — lazy, set when each loader succeeds; 999+ cap; hidden when 0), per-tab explainer (TAB_GUIDES map, refreshed in the tab click handler + switchTab), static Tip of the day (rotates by day-of-month). Right rail: Live activity feed (pushActivity, severity dots ok/info/warn/err, cap 20, honest empty state — fed from pollJob progress_message ticks + failed/broken alerts on loadLogs), chat dock (#chat-dock): the SAME #chat-widget node is DOM-moved between the rail and document.body by applyRailMode() on a (min-width:1360px) matchMedia change — single instance, panel auto-opens when docked unless the user closed it (chatUserClosed), floats as before below 1360), quick links (Report / SEO Insights / Competitors / Quality / Schedules / Settings / New Analysis). body.app-wide toggled for docked-chat CSS.
  • Crawl screen: progress card gains phase chips (Queued → Crawling → Analyzing → Done; updateCrawlPhase from job.status + current_stage; done chips for passed stages, .fail on Analyzing when failed), live stats strip (#crawl-stats: pages crawled = real pages_crawled, pages/min = derived from tick deltas, elapsed = client-side timer from first tick, last URL = parsed from progress_message only when it contains a real URL, else "—"), and a 2-col .progress-info zone below (static "What the audit checks" checklist from the landing copy + 5-item static SEO FAQ accordion: partial audits, not-measured checks, timings, redirects, competitor block). #progress-pages element removed (n/a to pollJob/showProgress). Crawl screen verified mid-run on example.org (stages through site_health in ~41s; queued → analyzing → done transitions observed; zero console errors).
  • Per-tab polish: Overview content-type cards are clickable (jump to Content tab
    • set the type filter; only when the type is a valid filter option and job open); Quality tab gains Expand All / Collapse All buttons (toggle open on #quality-content .audit-card); SEO Insights Overview History gains a sparkline of the monthly traffic_sum series (sparkline() helper, inline SVG, only when

    =2 numeric points, else honest "—"); rail nav badges wired into loadPages / loadContent / loadActions / loadSites / loadSchedules / loadLogs / loadCompetitors / loadQuality success paths.

  • Dead Chart.js CDN tag removed from index.html (was never used — zero Chart/analytics refs remain in frontend/).
  • Verified via Playwright (port 8001): 1440px — both rails visible, 13 rail nav entries, job card + ring, badges populated/updating (pages 261, content 807, actions 999+, sites 4), chat docked inside right rail (bounding-box check, panel fills slot), tab explainer + rail active state for all 13 tabs, quality expand/collapse (15 cards), theme toggle flips body bg (245,247,251 ↔ 11,18,32), no horizontal overflow; 375px — rails hidden, chat floats, crawl dashboard (4 stats, 2 info zones) no overflow; reduced-motion landing renders; zero console errors everywhere (checked 3 viewports). Layout geometry checked programmatically (this model cannot view screenshots). Test jobs (example.com/org/net) hard-deleted after verification; Sites list back to zuigo.ai + fluidcontrols.com × 2.
  • Deferred (in "Next up"): keyboard shortcuts; deeper per-tab widgets (ratio rings, run-now on schedules, status-chip filters on Pages/Links) — skipped to stay frontend-only + honest with data shapes that exist today.

• REAL BUG FIXED (found mid-run): crawler.py::fetch_page_text referenced undefined REQUEST_TIMEOUT (NameError) — every plain-fetch fallback crashed, so ALL competitor crawls died with "Competitor crawl returned no pages" and the first batch (5x) errored out. Fixed: REQUEST_TIMEOUT = 15 module constant.

• Timed-out giant crawls now produce HONEST partial reports instead of dead errors: audit_competitors outer wait was the same 1200s as the crawl cap, so the whole audit died (and deleted the ephemeral job) right when a big crawl hit its cap. Now competitor_audit_headroom_seconds = 2400 (config) extends the outer budget to crawl-cap + analysis headroom; the inner crawl cap still interrupts at 1200s and _analyze_one proceeds with the crawled slice (status: partial, pages_analyzed, "[crawl] timed out ... partial data").

• Re-ran competitor analyses on 6294eef9 (fluidcontrols.com) after clearing all rows: swagelok.com completed (400 pages), parker.com blocked (robots 403), tu-lok.com partial (163 pages, crawl cap), divineengg.in completed (68), stauff.com completed (400). /report returns all 5; SEO Insights tracked competitors = exactly these 5 (CACHE_VERSION 14 -> 15). Playwright: 5 cards, zero console errors.

• Ops note: after a worker kickstart arq may re-queue interrupted tasks — flush arq:* redis keys before re-enqueueing to avoid duplicate concurrent audits.

• (Earlier: truthfulness round committed d9690ce; report/links round fe8c9a5.)

• Links tab redirects made honest: "Ends in redirect (301/302)" (true 3xx bucket, was 0 for live job) split from "Followed redirects" (links that redirected then resolved, was 152 — previously shown under a "301/302" label). All Links table gains a "Redirected to" column showing final_url when redirect_count>0 (tooltip = full chain).

• Unchecked links explained: /api/links/{job}/all returns unchecked_count; frontend always renders an amber note (count + "recovered from crawl, not yet verified — run Check Links") whenever >0, regardless of which 200-row page is shown (was: note only when the current page happened to contain one).

• SEO Insights keyword cleanup: keyword_engine.MODIFIERS drops merchant-tone terms (price/oem/for sale/wholesale/exporter), variants capped at 3 per stem (MAX_MODIFIERS_PER_PHRASE), overview added to GREEDY_SLUGS + TITLE_STOP; CACHE_VERSION 11 -> 13. Live: site terms are now union/union manufacturer/union supplier/union manufacturer india + railway family; no "for sale" junk.

• SEO Insights competitors: SE Ranking rows sorted by domain_relevance desc; tracked/organic entries with zero overlap render an amber "No keyword-overlap data — not verified as an organic competitor; check the Competitors tab" note instead of looking like verified matches.

• Duplicates evidence: Site Health "Duplicate Content & Canonicals" card now lists the actual duplicate groups (URL pairs linkified, similarity label, cap 5 per group) via a new optional evidence block in auditCard.

• 349/349 tests (4 new: unchecked_count, merchant-tone drop + 3-cap, competitor relevance sort), node --check clean, live-verified on 6294eef9 (Playwright: both redirect cards, REDIRECTED TO column, amber unchecked note w/ 77, duplicate groups (59) evidence, no merchant keywords, no-overlap chips, zero console errors). (Earlier: report/links round committed fe8c9a5.)

• Report tab: standalone Indexation block and the Image-Optimization "Occurrences" stat card removed (image audit keeps Unique / WebP-AVIF / Without-dimensions / lazy counts; chat verdict no longer cites "occurrences"). Flow Count now means TRUE distinct funnel pages per target (depth 2 = unique intermediate_url, depth 1 = unique start_url) via $addToSet aggregation in backend/services/user_flow.py; detect_user_flows re-run on live job 6294eef9 (12,172 flows, top targets now 101 distinct vs fake 1429 before).

• Links tab: external links are now actually visible. Root cause: old link_health rows carry no external field. New scripts/backfill_link_external.py inserts missing external targets as status:"unchecked" rows (ran for 6294eef9 and 1c460760: +77 external rows each, 533->610 total; DB rows keep the flag, so the fallback is only for jobs never backfilled). Route fallback in links.py all_links: when zero flagged rows exist, derives the external set from page_links.external_link_urls (normalize_url) and re-filters; external=true API + filter dropdown + gray "Unchecked" badge + "Run Check Links" hint for stale rows. 346/346 tests, node --check clean, live-verified on 6294eef9 (external=true -> 77 rows, 0 console errors). (Earlier round: Actions dedupe with per-issue groups + batch issue_key, committed 94bb3fb.)

• Round (latest): blocked competitors now yield full partial reports

• UI/backend overhaul round (full suite 342/342, node --check clean, live-verified on 6294eef9):

  • Pages tab: Links column REMOVED (crawler summary still stores counts).
  • Links tab: full collapsible rows (details), external-link filter option, external badge + explainer; "Link Occurrences" card kept honest.
  • Contributors of SEO Actions: Expand All / Collapse All buttons (#expand-all-actions-btn/#collapse-all-actions-btn); each action card now a <details> with evidence linkified + capped at 8 items.
  • Analytics tab REMOVED entirely (nav, #tab-analytics block, Chart.js CDN, all chart helpers renderSingleAnalytics/renderAnalyticsCharts/ chartInstances/destroyCharts etc.; zero Chart/analytics strings in frontend/; VALID_TABS updated).
  • Alerts tab: Audit Trail section + audit-event-filter REMOVED; loadLogs renders alerts only.
  • SEO Insights: every section is now <details class="insights-section" open> + "How to use SEO Insights" explainer at top + per-section explainers (keywords, SERP); image accessibility card group (unique images, missing-alt, alt coverage %, pages with images) with accurate per-image counts from image_optimization audit; renderCompetitors shows tracked competitor cards (from competitor_gap_analyses, merged via external_insights with filters on job_id+status+domain) with "tracked" badge, status badge (partial/blocked), shared_keywords chips (capped 8); loadTrackedCompetitors + #tracked-competitors-list removed.
  • REAL BUGS FIXED (found by new tests): external_insights.py competitor merge crashed on missing get_db import (silently swallowed, so tracked competitors never merged) and dedupe seeded from se_names (filtered out every SE Ranking lookalike; now tracked_names only).
  • Keywords section merged smart site-derived terms: fetch_all_insights prepends get_smart_keywords(use_llm=False) output (flagged site_derived: True); frontend renders "site term" chip for them. routes/seo_insights.py CACHE_VERSION 5 -> 11 (multiple bumps).
  • keyword_engine.py::slug_phrases noise filters: strip file extensions (.pdf .mp4 ...), skip numeric-part-number leaves (e.g. /pn-03/ -> "pn"), skip single-word phrases that are just modifiers (india/price/oem), skip slugs ending in a generic section word, GREEDY_SLUGS extended (randd/rnd/uk/usa/awards/certifications/quality/team/careers/clients/ partners/gallery/downloads/videos/manuals/tenders/csr/...). Live keywords now exact-match product terms ("union manufacturer india", "double ferrule fittings", ...) + SE Ranking volume terms.
  • Reliability pack for competitor scans (HTTP-first crawl, 400-page cap, 20-min/competitor timeout -> partial status with error text, blocked on robots-403, stale-row recovery, frontend poll 60 min) shipped previously; live job shows parker.com blocked, swagelok re-crawl, status badges + Retry buttons.
  • New tests: backend/tests/test_ui_overhaul_round.py (15) — slug_phrases modifiers/part-numbers/extensions/generic-tail; keyword_engine smart merge in seo-insights route; fetch_all_insights competitors merge/dedupe/degradation (partial rows, blocked), keywords site terms prepend + no-dup; local_insights/image_optimization host patched via module attr (bound import) not just backend.db.mongo.get_db.
  • Live-verified via API + Playwright (seo-insights: 12 site-term chips + 15 keyword cards + 9 collapsible sections; competitors: 12 cards with tracked badges, 93 gap cards + Retry btns, zero console errors).

• Round (latest): blocked competitors now yield full partial reports (committed 4bfed03 recapped below). Bot-blocked competitors (≤1 page crawl, e.g. parker.com) produce a FULL partial report instead of being skipped after the status was set:

  • _analyze_one runs the shared SE-Ranking+SERP block first (baseline + _serp_keyword_gaps/_serp_features_gap/_backlink_gap via cached target keywords from extract_keywords_from_content) and only then branches: blocked → stub crawl-gap dicts with "Crawl blocked; ... not measured" errors; else → full crawl chain (analyze_pages/check_links/ compute_site_health/fetch_performance/audit_structured_data/content-gap dicts). se_rich (opportunity score + recommendations) computed in both paths; result status blocked when blocked.
  • config.py gains competitor_timeout_seconds: int = 1800; run_one uses it (30-min per-competitor cap; error message now "...(30-minute limit)").
  • GET /api/competitors/{job}/report accepts status {"$in": ["completed", "blocked"]} so the sectioned JSON report includes blocked competitors.
  • Frontend renderCompetitorGaps: blocked rows render the full report card with an amber "Blocked crawl — partial report" badge + note ("report below is SE Ranking + SERP data only") + Retry button (was: dead-end error card).
  • Tests test_competitor_blocked.py now 4 (patched fixture mocks _target_baseline/_se_rich_gap/_serp_*/_backlink_gap+ extract_keywords import; blocked tests assert se_rich + keyword/backlink gaps survive; new report-route test with$in` FakeDb.find support). Full suite 327/327.
  • Live-verified on 6294eef9-1f42-4974-babb-170165f7736f: parker.com → blocked with se_rich opp 100, backlink_gap domains, traffic/authority sections; Swagelok re-crawling (will hit 30-min cap legitimately); Playwright verified badge + Retry on the live card, zero console errors.
  • Committed as 4bfed03; pushed.

• Round (earlier): competitor audits that "silently completed" ... (previous status below) — blocked-status fix committed 8a0fa7e (marked blocked with errors, early return before SERP/SE spend, $unset stale errors on rerun, blocked card with Retry).

• Round (earlier): Competitor analyses were stalling in running status when a single large site (e.g. swagelok.com 1,000 pages) took over 30 mins, holding faster competitors (parker.com, hydac.com) blocked in asyncio.gather memory without writing their completed status to MongoDB.

  • Fix 1 (Incremental Persistence): _analyze_one and run_one now immediately persist each competitor's completed result to MongoDB as soon as it finishes, allowing frontend polling to surface completed competitors immediately.
  • Fix 2 (15-min Per-Competitor Timeout): Wrapped _analyze_one calls in asyncio.wait_for(..., timeout=900). If a competitor exceeds 15 minutes, it is cleanly saved with status: "error" without blocking the rest of the batch.
  • Fix 3 (DB Cleanup): Cleaned up stale running competitor rows and ephemeral jobs from past worker restarts.
  • Tests: Full suite 307/307 passing.

• Committed last as 412d33d; prior round pushed as 1cd70c4 (competitor gap report); this round pushed as d8bf1fc.

SE Ranking competitor gap report (this round)

• competitor_audit.py::_se_rich_gap(target_domain, comp_domain, target_job_id): reads TARGET's cached seo_insights_cache (no extra SE Ranking spend) and fetches COMPETITOR live best-effort per-key (domain_overview, domain_keywords(50), ranked_keywords(50), backlink_summary, domain_overview_history, backlink_list(50), keyword_gap); per-key errors collected. Computes keyword_analysis (shared / missing_from_target / unique_target / missing_detail [volume+cpc+difficulty] / shared_detail [target vs comp vol/cpc/KD] / top_opportunities = competitor's highest-volume keywords target misses), traffic_analysis (incl. traffic_value_estimate = SUM volume×CPC over comp domain_keywords), backlink_analysis (+ high_authority_sources domain_inlink_rank<20), authority_analysis (domain/page rank deltas). _opportunity_score (0-100: kw-opportunity 0.4 · traffic-growth 0.3 · RD-delta 0.3; None when no comparable metrics) + _build_recommendations (rule-based, high/med/low sorted).
• Wired into _analyze_one → stored as se_rich in competitor_gap_analyses; failures caught & logged (doc still writes).
• build_competitor_report(row) assembles named sections (executive_overview, keyword, traffic, content, backlink, authority, serp, on_page, ux, schema, technical, insights, source_report); empty sections stay honest. New GET /api/competitors/{target_job_id}/report (completed only, registered BEFORE /{target_job_id}/{competitor} so route wins).
• Frontend renderCompetitorGaps rewritten as a professional sectioned report per competitor: KPI strip (opportunity badge, pages, traffic "you → them", keywords, gap count), Keyword Analysis table (opportunity volume/CPC/difficulty + shared overlap table + SERP chips), Traffic table, Backlinks & Authority table (you lead / competitor ahead / tie), high-authority source chips + SERP-derived sources, Content + schema lists, SERP features, Technical/On-page/UX delta grids, Insights & Recommendations. Helpers: _fmtNum, _fmtMoney, _oppBadge, _pairRow, _deltaCard, _deltaGrid.
• Live-verified on job d14e24df (fluidcontrols.com) competitor hydac.com re-crawl: opp 100, traffic 141→4628, traffic value 54.2k, backlinks delta 118k, DR 46/78, 5 recs. Playwright zero console errors.
• Tests backend/tests/test_competitor_gap_report.py (6 new): _se_rich_gap assembly with FakeDb + monkeypatched se ranking fns, per-key degradation (7 errors), traffic_value_estimate, opp+recs, report assembler, empty row.
• All prior rounds below (SE Ranking sole paid provider, etc.) unchanged.
  • Cannibalization fully removed (rg -ri cannibal backend/ frontend/ = 0): deleted services/cannibalization.py; pipeline stage, summary key, quality route, site_health block, exec_summary entries, chat/sites refs, frontend stats card/report section/quality card all removed.
  • logsEl is not defined fixed: dead logsEl/appLogs refs removed (rg -n logsEl\|appLogs = 0); loadLogs catch renders into alertsEl.
  • Analytics tab handles no-study, empty history, single-analysis KPI cards, multi-analysis charts, keyword charts only with >=2 points.
  • Competitors: parallel batches Semaphore(2), cancel resets running→queued, dedupe enqueue, 30-min frontend poll cap
    (competitorPollDeadline), running cards surface errors.
  • Exact content counts: content_items unique per (job_id, content_type, source_url); ensure_indexes unique index in db/mongo.py (degrades to non-unique on duplicate); classifier dedupes per resolved URL, skips data: URIs, <source> only inside <picture>; image_optimization uses same URL-resolved dedupe so total_images == content_breakdown.image; crawler dedupes video_embeds by download URL + DuplicateKeyError guard. scripts/dedupe_content_items.py one-time migration for existing jobs (dedupes content_items + orphans content_extractions).
  • Report: raw "SEO Action Items" section + seo_action_items/action_items_list JSON key removed (report kept Quick Wins / Long-Term Improvements / Content Versions / Overview + KPI bars); action_items var is still fetched into _report_html for exec/recommendation rendering. Evidence guarantee lives in seo_analyzer._make_action, which returns None (skips with a warning) whenever check.evidence is empty, so every stored action has real evidence; Actions tab surfaces the Evidence block per action or a red "No evidence recorded" note.
  • Crawl speed: link_checker CHECK_CONCURRENCY 10→20, REQUEST_TIMEOUT 12→7, geo_alignment parallel via Semaphore(8), PSI sample cap settings.psi_page_sample = 20 (was all pages), mobile sample cap settings.mobile_sample_pages = 20 sorted by smallest click_depth, classifier/crawler/image_opt dedupe. Live fluidcontrols.com (261 pg) crawl time cut substantially; reports still render identical content.
  • Email reports: SMTP config DB app_settings key smtp first (GET/PUT /api/settings/smtp, masked), .env fallback (smtp_host/port/user/ password/from/use_tls in config.py + .env.example); notifications.py get_smtp_config/send_email (smtplib in asyncio.to_thread)/ email_report(job_id, to) builds branded PDF via reports _render_pdf; POST /api/reports/{job_id}/email; Analyze form optional email field (success + failure alert emails); Settings tab SMTP section; Report tab "Email report" button. Without SMTP configured the endpoint 502s with a clear message (tested live).
  • Crawl-progress live view: crawler writes current_stage/ pages_crawled into the job doc on every progress tick; progress card shows animated %, "Live" pulse dot, stage title (Crawling... / Running audit...), and "N pages crawled" line.
• SE Ranking is the sole paid insight provider (DataForSEO removed) — see below.
• DataForSEO removal + SE Ranking expansion round (committed, was working tree; full suite 285/285):
  • services/dataforseo.py and services/competitor_gap.py DELETED. SE Ranking (api.seranking.com/v1) is now the sole paid insights provider for: domain overview + organic traffic estimates, organic keywords, paid keywords (domain/overview/db), keyword gap (domain/keywords/comparison diff=1), competitors (domain/competitors), backlink overview (backlinks/summary), backlink sources (backlinks/all per_domain=1 ordered by domain_inlink_rank), referring domains (backlinks/refdomains), anchors (backlinks/anchors), top pages (backlinks/indexed-pages), page/domain authority (backlinks/authority inlink_rank/domain_inlink_rank), authority history (backlinks/authority/domain/history by_month), new/lost backlinks (backlinks/history -> new_lost_backlinks, backlinks/history/count -> new_lost_backlinks_count). Region source param is added for domain/* endpoints only (NOT backlinks/*). 100 credits per domain request.
  • NEW services/external_insights.py: fetch_all_insights(domain, job_id) orchestrator (se-ranking -> local per section; onpage = local crawl only; best-effort extras with *_error keys; GSC override block; SERP via serp_api with se_ranking.ranked_keywords fallback) + merge_gsc_into_insights moved verbatim from dataforseo.py. routes/seo_insights.py CACHE_VERSION 3 -> 4.
  • config.py: dropped dataforseo_login/password/cost vars; spend_tracker.py: dropped "dataforseo" rate. backlinks.py::fetch_backlinks: SE Ranking backlink_list (limit 100, source_api="se-ranking") first, SERP link: fallback.
  • Frontend: all DataForSEO strings gone (insightErrorKind titles, credits/ disabled error blocks, sourceLabel map, Labs sample note). SEO Insights gains: Overview History table, Top Organic Competitors cards, Backlink Profile section (authority cards, referring domains, anchor texts, top pages, authority history, new/lost + daily counts tables). Settings text now describes SE Ranking as the sole provider.
  • reports.py: methodology rows now name SE Ranking + competitors/overview history rows; Backlinks card gains profile rows (authority, top refdomains, anchors, top pages, new/lost); Domain Insights gains overview history, competitors and authority history tables (td.subhead style added).
  • Tests: test_dataforseo_resilience.py DELETED (11); test_se_ranking.py rewritten (26 tests: retry/hints/mapping for all new endpoints + external_insights chain tests); test_gsc.py imports merge_gsc_into_insights from external_insights. Suite 283 -> 285.
  • README.md updated (features, services list, env table); ServiceError + competitors.py docstrings cleaned. rg -ri dataforseo backend/ frontend/ = 0.
  • LIVE VERIFY PENDING (see Next up): launchd restart + refresh job 24fd33f6.
• programmatic-seo + ai-seo skill integration round (working tree; full suite 283/283):
  • Skills installed: ~/.agents/skills/programmatic-seo (SKILL.md v2.0.0) and ~/.agents/skills/ai-seo (SKILL.md v2.2.0) via npx skills add coreyhaines31/marketingskills --skill <name> -a opencode -g --copy -y (interactive TUI hangs; the -a opencode -g --copy -y flags make it non-interactive).
  • services/programmatic_seo.py (NEW): url_pattern(url) collapses the leaf path segment (and any numbers / ≥8-hex / >60-char segments) to {slug} (root and <2-segment paths -> None); detect_clusters(pages) groups 3+ pages sharing a pattern; audit_programmatic_seo(job_id) upserts programmatic_seo_audits — score + 4×25 subscores (structure/content_uniqueness/ internal_linking/indexation), checks (cluster detected, thin <150 words, near-duplicates via duplicate_content.duplicate_groups, internal linking via orphan_pages, sitemap coverage), clusters rows (pattern/page_count/thin_pages/ duplicate_pages/unlinked_pages/sample_urls), template_pages/total_pages/ template_page_share/thin/duplicate/unlinked/duplicate_title/not_indexable counts.
  • services/ai_visibility.py extended IN PLACE (kept 4×25 subscore model; new signals land as checks + summary keys): TRAINING_ONLY_AGENTS={"ccbot"}, ANSWER_BLOCK_WORDS/DEFINITION_BLOCK_WORDS/FRESHNESS_DAYS=183; new probes /pricing.md, /pricing.txt, /okf/; page scan up to MAX_SCAN_PAGES=50 with _ai_extractability(html) -> answer_block/ definition_first/faq_heading/comparison_table/stat_cited/author/fresh/ semantic_landmark keys; new summary keys (pricing_md_present, pricing_txt_present, okf_present, scanned_pages, answer_block_pages, definition_first_pages, faq_heading_pages, comparison_table_pages, stat_cited_pages, author_pages, fresh_pages, semantic_landmark_pages, blocked_training_agents) + 4 new checks (machine-readable pricing, answer blocks 0.3 ratio, author 0.3 ratio, freshness 0.3 ratio) + CCBot-blocked info check; fixed ai_extract->ai_ext NameError.
  • Pipeline: wave2 _stage("programmatic_seo", ...) in analysis.py; summary dict gains programmatic_seo.
  • site_health merge: pseo metrics (template_clusters/template_pages/ thin_template_pages/duplicate_template_pages/unlinked_template_pages/ programmatic_seo_score) + medium/high issues; ai_visibility merge extended with pricing_md / author issues.
  • exec_summary: keys programmatic_thin, programmatic_duplicates, programmatic_linking, ai_pricing_md, ai_eaat_signals in EFFORT/TITLES/ EXPLANATIONS/HOW_TO_FIX/ISSUE_KEY_FROM_MESSAGE + _evidence_for branches.
  • chat_service: PROGRAMMATIC_SEO_GUIDANCE + AI_SEO_GUIDANCE constants (vendored from the skills, marketing-skills, MIT) wired into SYSTEM_PROMPT / GENERAL_SYSTEM_PROMPT / FULL_SITE_PROMPT section list; _programmatic_seo_context
    • _ai_seo_context builders; aliases programmatic-seo/programmatic/templates/ template and ai-seo/ai-visibility/ai_search/ai.
  • quality.py: GET /{job}/programmatic-seo. sites.py hard-delete list extended (programmatic_seo_audits, sitemap_audits, ai_visibility_summaries, local_seo_summaries, cannibalization_summaries, hreflang_audits, url_hygiene_audits, indexation_audits, image_optimization_audits, exec_summaries, serp_cache).
  • reports.py: programmatic_html section (score + 4 subscore bars + cluster table + checks) appended to _report_html; ai_html extended with blocked agents + machine-readable files row + scanned/author/freshness line.
  • app.js Report tab: AI-Search Visibility block adds AI files row + author attribution + scanned-page note + training-crawler note; new Programmatic SEO block (3-col stats + 4 s

*Truncated - read the full file at https://github.com/ZuiGO/ZuiGo-Engine/blob/2a1923842fd57617b51a838c15f65d5d470e3427/AGENTS.md.