LLMFeeder

Imported from jatinkrmalik/LLMFeeder (AGENTS.md). Install upstream with npx skills add jatinkrmalik/LLMFeeder. Copyright stays with the author.

LLMFeeder - AI Agent & Contributor Guide

This guide is designed for AI agents (like Claude, ChatGPT, etc.) and human contributors working on the LLMFeeder browser extension. It provides a comprehensive reference for understanding the codebase, development workflow, and testing procedures.

Table of Contents

1. Quick Reference
2. Repository Structure
3. Development Workflow
4. Extension Architecture
5. Local Testing Guide
6. Debug Mode & Logging
7. Common Development Tasks
8. Build & Release Process
9. CI/CD Pipeline

Quick Reference

# Build extensions
./scripts/build.sh chrome    # Chrome only
./scripts/build.sh firefox   # Firefox only
./scripts/build.sh all       # All packages
make chrome                  # Alternative: using Make

# Start local test server
python3 -m http.server 8080

# Git workflow
git checkout -b feature/your-feature-name
# ... make changes ...
git add extension/
git commit -m "feat: description"
git push origin feature/your-feature-name

Key Files for Feature Development:

• extension/content.js - Core conversion logic (main content extraction, markdown)
• extension/popup.js - Popup UI and settings handling
• extension/popup.html - UI structure
• extension/styles.css - Styling
• extension/manifest.json - Extension configuration

Testing Files:

• testbench.html - Comprehensive test page
• iframe-content-test.html - Nested iframe content for testing

Repository Structure

LLMFeeder/
├── extension/                 # Browser extension source
│   ├── icons/                # Extension icons (16, 48, 128px)
│   ├── libs/                 # Third-party libraries
│   │   ├── readability.js    # Mozilla content extraction
│   │   ├── turndown.js       # HTML to Markdown
│   │   └── browser-polyfill.js # Cross-browser compatibility
│   ├── manifest.json         # Extension manifest (MV3 base)
│   ├── popup.html            # Popup UI
│   ├── popup.js              # Popup logic (18KB)
│   ├── styles.css            # Popup styling
│   ├── content.js            # Content script (42KB) - CORE LOGIC
│   └── background.js         # Background script for keyboard shortcuts
│
├── scripts/
│   └── build.sh              # Build script (240 lines)
│
├── .github/workflows/
│   ├── pr-validation.yml     # PR build validation
│   └── release.yml           # Release asset upload
│
├── testbench.html             # Main testing page
├── iframe-content-test.html   # Nested iframe test content
├── Makefile                   # Build automation
└── README.md                  # User documentation

Development Workflow

Step 1: Create a Feature Branch

git checkout main
git pull origin main
git checkout -b feature/your-feature-name

Step 2: Make Changes

Edit files in the extension/ directory based on your feature:

Step 3: Build and Test Locally

# Build Chrome extension
./scripts/build.sh chrome

# Extract for local testing
unzip -q dist/LLMFeeder-Chrome-v*.zip -d /tmp/llmfeeder-test
# OR just use the chrome-unpacked directory if created

Step 4: Load in Browser

1. Open chrome://extensions/
2. Enable "Developer mode"
3. Click "Load unpacked"
4. Select the extracted extension directory (or extension/ folder directly)

Step 5: Test with Test Bench

# In project root
python3 -m http.server 8080

Then navigate to http://localhost:8080/testbench.html and test your changes.

Step 6: Commit and Push

git add extension/
git commit -m "feat: description of changes"
git push origin feature/your-feature-name

Extension Architecture

Component Communication

┌─────────────────┐     messages     ┌─────────────────┐
│   popup.js      │ ───────────────▶ │   content.js    │
│  (UI/Settings)  │                  │ (Core Logic)    │
└─────────────────┘                  └─────────────────┘
        ▲                                    │
        │                                    │
        │ browser storage                   │
        │                                    ▼
┌─────────────────┐              ┌─────────────────┐
│ browser.storage │              │   Turndown.js   │
│  (sync prefs)   │              │  (HTML→MD)      │
└─────────────────┘              └─────────────────┘
                                         ▲
                                         │
┌─────────────────┐                      │
│ background.js   │ ─────────────────────┘
│ (shortcuts)     │
└─────────────────┘

Content Script (content.js) - Core Functions

Popup Script (popup.js) - Key Functions

Message Actions (for communication between scripts)

Local Testing Guide

Quick Test Cycle

# 1. Build extension
./scripts/build.sh chrome

# 2. Extract to temp location
mkdir -p /tmp/llmfeeder-test
unzip -q -o dist/LLMFeeder-Chrome-v*.zip -d /tmp/llmfeeder-test

# 3. Start test server
python3 -m http.server 8080

Test Bench Scenarios

Open http://localhost:8080/testbench.html and test:

Test Settings to Use

When testing with the test bench:

1. Enable Debug Mode (gear icon → Debug Mode checkbox)
2. Enable "Preserve Tables" for table tests
3. Set Content Scope to "Full Page" for iframe tests
4. Set Content Scope to "Main Content" for article tests

After Testing

1. Click "Copy Logs" button in settings to get debug output
2. Paste logs in your PR or issue for troubleshooting
3. Click "Remove" on the extension in chrome://extensions/
4. Reload extension after code changes (click reload icon)

Debug Mode & Logging

Enabling Debug Mode

1. Open LLMFeeder popup
2. Click gear icon (Settings)
3. Check "Debug Mode" checkbox
4. Logs are now collected during conversions

Accessing Debug Logs

1. After running a conversion, go to Settings
2. Click "Copy Logs" button
3. Logs are copied to clipboard
4. Paste in issue/PR for debugging

Debug Log Format

[2025-01-26T12:34:56.789Z] Conversion started
{
  "contentScope": "mainContent",
  "preserveTables": true,
  "includeImages": true,
  "includeTitle": true
}
[2025-01-26T12:34:56.890Z] Content extracted
{
  "innerHTMLLength": 45678
}
[2025-01-26T12:34:57.123Z] Conversion successful
{
  "markdownLength": 12345,
  "hasTables": true
}

Adding Debug Logs

In content.js, use the DebugLog object:

DebugLog.log('Your message here', { key: 'value' });
DebugLog.error('Error message', errorObject);

Common Development Tasks

Adding a New Setting

1. Add to popup.html - Create UI element (checkbox, radio, input)
2. Add to popup.js - Add to loadSettings() and saveSettings()
3. Add to content.js - Use in convertToMarkdown() function

// In popup.js
const myNewSetting = document.getElementById("myNewSetting");

// In loadSettings()
myNewSetting.checked = data.myNewSetting || false;

// In saveSettings()
const myNewSetting = myNewSetting.checked;
await browserAPI.storage.sync.set({ myNewSetting });

// In content.js convertToMarkdown()
const myNewSetting = settings.myNewSetting || false;

Adding a New Keyboard Shortcut

1. Edit manifest.json - Add to commands section:

"my_new_command": {
  "suggested_key": {
    "default": "Alt+Shift+N",
    "mac": "Alt+Shift+N"
  },
  "description": "My new command description"
}

2. Edit background.js - Add handler:

chrome.commands.onCommand.addListener((command) => {
  if (command === 'my_new_command') {
    // Your logic here
  }
});

Adding Custom Markdown Conversion Rules

In content.js, modify configureTurndownService():

turndownService.addRule('myCustomRule', {
  filter: 'myElement',
  replacement: function(content, node) {
    // Return markdown representation
    return '**Custom:** ' + content;
  }
});

Modifying Table Handling

Tables are handled in configureTurndownService() when settings.preserveTables is true. Key rules:

• table - Wraps content in newlines
• tableRow - Creates pipe-separated rows with header separator
• tableCell - Formats individual cells
• thead/tbody - Prevent extra newlines

Build & Release Process

Build System Overview

The project uses two build methods:

1. Shell Script (scripts/build.sh) - Full-featured build
2. Makefile - Convenient targets

Build Script Usage

# Build all packages
./scripts/build.sh all

# Build specific package
./scripts/build.sh chrome
./scripts/build.sh firefox
./scripts/build.sh source

# Build with custom version
./scripts/build.sh --version 2.2.1 all

Output Files

Built packages are created in dist/:

• LLMFeeder-Chrome-v{version}.zip - Chrome extension
• LLMFeeder-Firefox-v{version}.zip - Firefox extension
• LLMFeeder-Source-v{version}.zip - Source package

Version Bumping

1. Edit extension/manifest.json:

   "version": "2.2.1"

2. Build with version:

   ./scripts/build.sh --version 2.2.1 all

CI/CD Pipeline

Pull Request Validation (.github/workflows/pr-validation.yml)

Triggered on:

• PR opened, synchronized, or reopened
• Push to main branch

Validates:

• Chrome package builds
• Firefox package builds
• Source package builds
• Manifest JSON validity

Release Workflow (.github/workflows/release.yml)

Triggered on:

• GitHub release creation

Actions:

• Extracts version from git tag
• Builds all packages with version
• Uploads assets to release

Creating a Release

# Tag and push
git tag v2.2.1
git push origin v2.2.1

# Then create release on GitHub
# CI will automatically build and upload packages

Browser Compatibility Notes

Chrome (Manifest V3)

• Uses service_worker in background
• Default for new features

Firefox (Manifest V2)

• Uses scripts array in background
• Build script automatically converts manifest

Cross-Browser Considerations

• Use browserAPI wrapper in popup.js
• Use browserRuntime wrapper in content.js
• Test on both browsers before submitting PR

Settings Reference

Metadata Template Variables

• {title} - Page title
• {url} - Page URL
• {date} - Published date
• {author} - Article author
• {siteName} - Site name
• {excerpt} - Article excerpt

Troubleshooting

Extension not loading

• Check manifest.json for syntax errors
• Verify all referenced files exist
• Check browser console for errors

Content extraction not working

• Enable debug mode
• Copy logs and check for errors
• Verify content scope setting

Tables not converting to Markdown

• Ensure "Preserve Tables" is enabled
• Check if HTML table structure is valid
• Review debug logs for conversion errors

Build script fails

• Ensure jq and zip are installed
• Make script executable: chmod +x ./scripts/build.sh
• Check file permissions in extension/ directory

Contributing Guidelines

1. Branch Naming: Use feature/, fix/, or docs/ prefixes
2. Commit Messages: Use conventional commits (feat:, fix:, docs:, etc.)
3. Testing: Always test with testbench.html before submitting
4. Debug Logs: Include debug logs when reporting issues
5. Co-Authored-By: Add when collaborating with AI agents

feat: add new feature description

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Quick Command Reference

# Development
make chrome                  # Build Chrome
make firefox                 # Build Firefox
make clean                   # Clean dist/
make help                    # Show help

# Testing
python3 -m http.server 8080  # Start test server
open http://localhost:8080/testbench.html

# Git
git checkout -b feature/name # New branch
git add extension/           # Stage changes
git commit -m "feat: desc"   # Commit
git push origin feature/name # Push

# Cleanup
lsof -ti:8080 | xargs kill   # Kill server on port 8080

External Libraries

These are located in extension/libs/ and should not be modified without careful consideration.

Cursor Cloud specific instructions

Durable notes for agents working in the Cursor Cloud VM. Standard commands live in the sections above; this section only captures non-obvious caveats.

• No package manager / no install step: there is no package.json, lockfile, or node_modules. Third-party libs are vendored in extension/libs/. The startup update script has nothing to install; it only ensures scripts/build.sh is executable.
• Toolchain is preinstalled in the VM: jq, zip, python3, make, Chrome (/usr/local/bin/google-chrome), and Firefox (/usr/local/bin/firefox → /opt/firefox, installed from Mozilla's tarball for cross-browser testing). Both browsers persist in the VM snapshot, so they are not part of the update script.
• There is no unit-test suite. The "lint/test" equivalent is the build + manifest-JSON validation done in CI (.github/workflows/pr-validation.yml): bash scripts/build.sh all then jq empty on the generated Chrome/Firefox manifests. Run those to validate changes.
• Build/run: bash scripts/build.sh chrome (or firefox/source/all). Output zips go to dist/ (gitignored).
• End-to-end test flow: extract the Chrome zip (e.g. unzip -o dist/LLMFeeder-Chrome-v*.zip -d /tmp/llmfeeder-chrome), load it unpacked at chrome://extensions/ (Developer mode → Load unpacked), start python3 -m http.server 8080 from repo root, open http://localhost:8080/testbench.html, then use the popup's "Convert & Copy" button.
• Prefer loading the built Chrome package (from dist/) rather than the raw extension/ folder: the source manifest.json is MV3 with browser_specific_settings/menus that Chrome ignores/warns on; build.sh strips those for a clean Chrome load. Firefox needs the built package because build.sh rewrites the background section.
• Firefox E2E: build (bash scripts/build.sh firefox), extract (unzip -o dist/LLMFeeder-Firefox-v*.zip -d /tmp/llmfeeder-firefox), launch firefox &, go to about:debugging#/runtime/this-firefox → "Load Temporary Add-on…" → pick /tmp/llmfeeder-firefox/manifest.json. The temp add-on has a temporary ID (expected). Both Chrome and Firefox conversions were verified end-to-end against testbench.html.
• Token counting fetches encodings from https://tiktoken.pages.dev/* but has an offline heuristic fallback and caches results, so core convert/copy/download works without network access.