feedreader

Imported from newscientist101/feedreader (AGENTS.md). Install upstream with npx skills add newscientist101/feedreader. Copyright stays with the author.

Agent Instructions

This is a Go web application — a multi-user feed reader — designed for standalone, self-hosted deployment.

See README.md for user-facing docs (features, API endpoints, scraper config, etc.). See STYLE.md for coding conventions — read it before writing code.

Important

Do not use subagents to read all files and output all file contents in full. This will fail due to context/token limits and should not be attempted.

Do NOT build, run, or test on this VM

This VM is no longer a build/validation environment. Do not run any build, compile, run, or test commands locally, including (non-exhaustive): make check, make build, make test, make lint, make vulncheck, make fmt-check, go build, go test, go run, npm ci, npm test, or starting the server. Do not install or upgrade toolchains for the purpose of building.

GitHub Actions CI is the sole build and validation authority. Validate changes by pushing to a branch and reading the CI workflow results, not by running anything locally. The make check steps documented below describe what CI runs — they are for reference, not for local execution.

You may still use read-only tooling (reading files, git status/diff/log, ripgrep/grep, etc.). Make your edits, commit, push, and rely on CI.

Tech Stack

• Go (module github.com/newscientist101/feedreader) with the standard library net/http router
• SQLite via modernc.org/sqlite (pure-Go, no CGO)
• sqlc for type-safe query generation (db/queries/ → db/dbgen/)
• HTML templates (html/template) served server-side
• Vanilla JS frontend (no framework) in srv/static/
• Runs on port 8000 (configurable via --port flag or config.toml)

Code Layout

cmd/srv/main.go          Entry point — parses flags, opens DB, starts server
srv/
  server.go              HTTP server, all route handlers (~2600 lines, the bulk of the app)
  auth.go                Pluggable auth middleware (AuthProvider interface; DEV=1 bypass)
  filter.go              Exclusion-rule filtering (keyword/author per folder)
  content_filter.go      Per-feed content transform filters
  category_tree.go       Nested category/folder tree builder
  retention.go           Data retention / old-article cleanup
  feeds/
    fetcher.go           RSS/Atom feed fetching with conditional GET support
    parser.go            RSS and Atom XML parser
    fetcher_test.go      Fetcher tests
    parser_test.go       Parser tests
  scrapers/
    runner.go            CSS-selector-based HTML scraper (goquery)
    runner_test.go       Scraper tests
    json_scraper.go      JSON API scraper
  sources/
    registry.go          Feed source registry interface and Resolve logic
    defaults.go          DefaultRegistry with all built-in sources
    steam.go             Steam news URL normalization and name lookup
    reddit.go            Reddit subreddit auto-naming
    huggingface.go       HuggingFace config-based name resolution
    *_test.go            Unit tests for each source and the registry
  huggingface/
    huggingface.go       Hugging Face-specific feed source
  opml/
    opml.go              OPML import/export
  email/
    email.go             Newsletter email receiving (SMTP)
  templates/             Server-rendered HTML templates
    base.html            Shared layout (nav, head, sidebar)
    index.html           Main article list view
    article.html         Single article view
    feeds.html           Feed management page
    scrapers.html        Scraper management page
    settings.html        User settings page
    queue.html           Reading queue page
    history.html         Reading history page
    category_settings.html  Folder exclusion-rule settings
  static/
    app.js               Entry point — imports modules, runs init (~360 lines)
    script.js            Service worker registration
    sw.js                Service worker
    style.css            All styles (~2900 lines)
    modules/             ES modules (each with a .test.js companion)
      api.js             Fetch wrapper for /api/* endpoints
      article-actions.js Mark read/unread, star, queue, auto-mark-read observer
      articles.js        Article rendering — cards, actions, embeds, user prefs
      counts.js          Unread/starred counts, feed status, error banners
      drag-drop.js       Drag-and-drop reordering for feeds and folders
      dropdown.js        Dropdown toggle and click-outside close
      feed-errors.js     Feed error banner (shared leaf: counts + feeds)
      feeds.js           Feed CRUD, load articles, edit modal
      folders.js         Folder CRUD, category settings page
      icons.js           SVG icon constant strings
      offline.js         Service worker, online/offline handling, cache
      opml.js            OPML import/export
      pagination.js      Cursor-based pagination and infinite scroll
      queue.js           Reading queue page init
      read-button.js     Read/unread toggle button (shared leaf: articles + article-actions)
      scraper-page.js    Scraper management page (tabs, AI, config modal)
      settings-page.js   Settings page init, cleanup, newsletter
      settings.js        User settings (get/save/apply)
      sidebar.js         Sidebar toggle, folder navigation, collapse
      timestamps.js      Tooltip initialization for timestamps
      utils.js           formatTimeAgo, stripHtml, truncateText, etc.
      views.js           View mode switching (cards/list/magazine/expanded)
db/
  db.go                  DB open, migrations, pragmas
  sqlc.yaml              sqlc configuration
  migrations/            Numbered SQL migration files (001–015)
  queries/               SQL query files for sqlc
  dbgen/                 Generated Go code from sqlc (do not edit by hand)

Key Patterns

• Authentication: All non-static routes go through auth middleware. Auth is handled by pluggable providers configured in config.toml. See srv/auth.go for the AuthProvider interface. Supported providers: proxy (generic header-based), tailscale, cloudflare, authelia, oauth2_proxy. Set DEV=1 to skip auth for local development.
• Database migrations: Auto-applied on startup in db.Open(). Add new migrations as sequentially numbered .sql files in db/migrations/.
• sqlc workflow: Edit SQL in db/queries/*.sql, then run go generate ./db/... to regenerate db/dbgen/.
• Single-file server: Most handler logic lives in srv/server.go. It's large but flat — each handler is a standalone function on the server struct.
• ES modules: Client-side JS uses native ES modules (no bundler). app.js is the entry point (<script type="module">), importing from modules/. An import map in base.html maps module URLs to versioned URLs (?v=hash) for cache busting. No inline onclick handlers — all event wiring uses data-action attributes with delegated addEventListener. Circular dependencies are resolved via late-bound setters (e.g., setArticleActionDeps()). Each module has a companion .test.js file.
• Read-state pipeline: db/articles.is_read is the authoritative source of truth. srv.CountsCache (per-user, 30-second TTL) caches /api/counts results and is invalidated on every mutating handler. The client-side read-queue module (srv/static/modules/read-queue.js) owns transient unsent read events with sessionStorage durability and keepalive flush (flushed on pagehide, visibilitychange→hidden, and before article navigation). The returnFromArticleList sessionStorage marker drives Back-navigation replay via a peek-then-consume-on-success pattern in spa-nav.js. /api/counts applies the same folder-exclusion filter as the listing endpoints so the unread badge always matches visible cards. The counts poll loop in counts.js refreshes every 60 seconds while the tab is visible and pauses when hidden.
• Build & validation: Performed by GitHub Actions CI, not on this VM. See the "Do NOT build, run, or test on this VM" rule above and the Build Workflow section below (reference only).
• Service: Optionally managed via systemd using the provided srv.service.example template.

Build Workflow

Reference only — do not run these locally. Builds and tests run in GitHub Actions CI (see the "Do NOT build, run, or test on this VM" rule above). The table below documents what CI executes. Validate your changes by pushing and checking the CI results.

CI runs all validation steps in order:

Other useful targets:

Linting details

• Go (.golangci.yml): errcheck, govet, staticcheck, gocritic (all checks enabled), misspell, nilerr, errorlint, bodyclose, ineffassign, unused. errcheck/bodyclose/unused are suppressed in test files; the std-error-handling exclusion preset has been removed. Generated db/dbgen/ is excluded entirely.
• JS (eslint.config.mjs): no-undef, no-unused-vars, eqeqeq, no-eval, etc. Functions called from HTML onclick attributes are whitelisted in varsIgnorePattern.
• CSS (.stylelintrc.json): stylelint with stylelint-config-standard. Catches duplicate selectors/properties, deprecated values, redundant shorthands, and pseudo-element notation. Stylistic color/alpha notation rules and no-descending-specificity are disabled.
• Templates (cmd/lint-templates/): Validates Go html/template files parse correctly with base.html, checks for mismatched {{ }} delimiters, unclosed HTML tags, mismatched open/close tags, and unused FuncMap entries.
• HTML (djlint, .djlintrc): HTML linting for template files with Go template profile. Configured to ignore H006 (img dimensions), H019 (javascript: hrefs), H021 (inline styles), H023 (entity refs), H031 (meta keywords).

Fixing formatting

If CI's fmt-check step fails, fix the formatting in your editor (matching goimports) and push again. Do not run make fmt/make check locally.

Viewing the App During Development

See README.md for full details. Quick options:

1. No auth (dev mode): Run with DEV=1 ./feedreader to skip auth and browse http://localhost:8000/ directly.
2. With config: Create a config.toml with an auth provider configured and run ./feedreader --config config.toml.

Test Users

All testing and development should use user 1 (external_id: dev-user-1, email: test@example.com) unless explicitly asked to use another user.

• Do not modify user 4 — this is the real production user.
• Do not create additional users.

Look up user credentials with:

sqlite3 db.sqlite3 "SELECT id, external_id, email FROM users;"

Issue Tracking with bd (beads)

All task tracking uses bd (beads). Do not use markdown TODOs or task lists. Issue IDs use the feedreader- prefix (e.g., feedreader-fc2).

The Dolt server runs as a systemd service (beads-dolt.service) on port 3307. A task agent (task-agent.service) runs hourly via systemd timer, picks ready tasks, and works them autonomously. The agent script and systemd units live in /home/exedev/ops/ (a local-only git repo, no remote).

A troubleshooter agent (troubleshooter.service) runs daily at 06:00 UTC. It scans the feedreader service logs for ERROR-level entries and crashes. If any are found (after filtering), it launches a Shelley conversation to investigate and create bd tasks. It does not fix issues — only investigates and files bugs.

Known-issues file (/home/exedev/ops/troubleshooter-known-issues.txt): One fixed-string pattern per line; lines starting with # are comments. Entries are automatically removed after one run. When fixing a bug that produces log errors, add a matching pattern here so the troubleshooter skips stale log lines from before the fix.

Finding and doing work

bd ready                              # Show unblocked issues
bd show <id>                          # Full details (description, design, notes, acceptance)
bd update <id> --claim                # Claim a task (sets in_progress + assignee)
# ... do the work, push, and confirm CI is green (do NOT build locally) ...
bd close <id> -r "what was done"       # Complete it

Creating issues

# Simple task
bd create "Fix pagination bug" -t bug -p 1 -d "Description of what's broken"

# Task discovered while working on another
bd create "Edge case in feed parser" -t bug --deps discovered-from:feedreader-fc2

# Batch from markdown file (## headings = issue titles, body = description)
bd create -f tasks.md

Types: bug, feature, task, epic, chore, decision Priorities: 0 (critical) through 4 (backlog), default 2

Planning with epics

For multi-step work, create an epic with children and dependencies:

# Create the epic
bd create "Redesign settings page" -t epic -d "Goals and rationale" \
  --design "Technical approach and architecture" \
  --acceptance "Definition of done"

# Create child tasks
bd create "Extract settings components" -t task --parent <epic-id>
bd create "Add validation" -t task --parent <epic-id>

# Set ordering via dependencies
bd dep add <validation-id> <extract-id>   # validation depends on extract

Epic fields:

• description — why this work exists, goals
• design — technical approach, architecture, module structure
• acceptance — concrete done-criteria
• notes — working memory, append with bd update <id> --append-notes "..."

To set long-form fields from a file:

cat plan.md | EDITOR="cp /dev/stdin" bd edit <id> --design

Dependencies and blocking

bd dep add <issue> <depends-on>       # issue is blocked by depends-on
bd blocked                            # Show all blocked issues
bd dep tree <id>                      # Visualize dependency graph

bd ready only shows issues with no open blockers.

Other useful commands

bd list                               # Open issues
bd list --all                         # Include closed
bd search "keyword"                   # Full-text search
bd children <epic-id>                 # Show child issues
bd comments add <id> "note"           # Add a comment
bd update <id> --append-notes "..."   # Append to working memory

Rules

• Track all work in bd — no markdown TODOs, no ad-hoc task lists
• Check bd ready before starting work
• Claim before working: bd update <id> --claim
• Validate via CI (push and check the workflow), never by building locally, before closing a task
• Link discovered work with --deps discovered-from:<id>
• Do NOT use bd edit without the stdin pipe pattern (it opens vim)

Landing the Plane (Session Completion)

When ending a work session, you MUST complete ALL steps below. Work is NOT complete until git push succeeds.

MANDATORY WORKFLOW:

1. File issues for remaining work - Create issues for anything that needs follow-up
2. Run quality gates via CI (if code changed) - push and confirm the GitHub Actions CI workflow passes; do not build/test on this VM
3. Update issue status - Close finished work, update in-progress items
4. PUSH TO REMOTE - This is MANDATORY:

   git pull --rebase
   git push
   git status  # MUST show "up to date with origin"

5. Clean up - Clear stashes, prune remote branches
6. Verify - All changes committed AND pushed
7. Hand off - Provide context for next session

CRITICAL RULES:

• Work is NOT complete until git push succeeds
• NEVER stop before pushing - that leaves work stranded locally
• NEVER say "ready to push when you are" - YOU must push
• If push fails, resolve and retry until it succeeds