data

Imported from docxology/literature (data/AGENTS.md). Install upstream with npx skills add docxology/literature --skill data. Copyright stays with the author.

data/ - Academic Literature Search & Management

Purpose

The data/ directory serves as the central repository for academic literature search, PDF storage, reference management, and research paper organization. It integrates with the infrastructure/literature/ module to provide a literature management workflow.

Directory Structure

data/
├── library.json          # JSON index of all papers with metadata
├── references.bib        # BibTeX bibliography file
├── summarization_progress.json # Summarization progress tracking (auto-generated)
├── failed_downloads.json # Failed PDF download attempts (for retry, auto-generated)
├── pdfs/                 # Downloaded PDFs (named by citation key) — created
│   │                     #   on demand by download operations; not present in
│   │                     #   the current checkout (verified 2026-08-29)
├── summaries/            # LLM-generated paper summaries
│   ├── smith2024machine_summary.md
│   ├── jones2023deep_summary.md
│   └── ...
├── extracted_text/       # Extracted text from PDFs
├── embeddings/           # Cached embedding files (JSON) for semantic analysis
├── output/               # Meta-analysis outputs and visualizations
├── AGENTS.md             # This detailed documentation
└── README.md             # Quick reference guide

Core Components

Library Index (library.json)

JSON database containing metadata for all indexed papers:

{
  "version": "1.0",
  "updated": "2025-12-02T04:42:16.615302",
  "count": 456,
  "entries": {
    "smith2024machine": {
      "citation_key": "smith2024machine",
      "title": "Machine Learning Advances in 2024",
      "authors": ["Dr. Jane Smith", "Dr. John Doe"],
      "year": 2024,
      "doi": "10.1234/example.doi",
      "source": "arxiv",
      "url": "http://arxiv.org/abs/2401.00001",
      "pdf_path": "data/pdfs/smith2024machine.pdf",
      "added_date": "2025-12-01T10:00:00.000000",
      "abstract": "This paper presents...",
      "venue": "arXiv preprint",
      "citation_count": 42
    }
  }
}

Features:

• Unique citation keys as primary identifiers
• Source tracking (arxiv, semanticscholar, crossref, pubmed)
• Download status via pdf_path field
• Metadata preservation
• Timestamp tracking for addition dates

Bibliography (references.bib)

Standard BibTeX format bibliography automatically generated from the library:

@article{smith2024machine,
  title={Machine Learning Advances in 2024},
  author={Smith, Jane and Doe, John},
  journal={arXiv preprint},
  year={2024},
  doi={10.1234/example.doi},
  url={http://arxiv.org/abs/2401.00001}
}

Integration:

• Can be manually copied to manuscript systems if needed
• Standard BibTeX format compatible with any LaTeX/BibTeX system
• Maintained independently in this repository

PDF Storage (pdfs/)

Downloaded PDFs organized by citation key:

Naming Convention:

• citation_key.pdf (matches library.json keys)
• Example: smith2024machine.pdf

Download Sources:

• Direct links from arXiv, Semantic Scholar
• Open access versions via Unpaywall fallback
• Legal sources only (no unauthorized access)

Failed Downloads Tracker (failed_downloads.json)

Automatically tracks failed PDF download attempts for retry capability:

File Format:

{
  "version": "1.0",
  "updated": "2025-12-12T10:00:00",
  "failures": {
    "smith2024paper": {
      "citation_key": "smith2024paper",
      "title": "Paper Title",
      "failure_reason": "network_error",
      "failure_message": "Connection timeout",
      "attempted_urls": ["url1", "url2"],
      "source": "arxiv",
      "timestamp": "2025-12-12T10:00:00",
      "retriable": true
    }
  }
}

Features:

• Automatic tracking of all download failures (except "no_pdf_url" which is just a warning)
• Retriable detection (network errors, timeouts are retriable; access denied, not found are not)
• Default skip behavior - Previously failed downloads are automatically skipped on subsequent runs
• Retry support via --retry-failed flag or interactive prompts
• Auto-cleanup when downloads succeed
• Integration with all download operations (workflow, meta-analysis, download-only)

Usage:

# Retry failed downloads
python3 scripts/literature_search.py --download-only --retry-failed

# Or use interactive prompts (automatically prompts if failures exist)
python3 scripts/literature_search.py --search

Summaries (summaries/)

AI-generated paper summaries using local LLM integration:

File Format: citation_key_summary.md

# Paper Summary: Machine Learning Advances in 2024

## Key Contributions
- Novel algorithm for X
- Improved performance on Y
- Theoretical analysis of Z

## Methodology
- Approach: Deep learning with transformers
- Datasets: Standard benchmarks (CIFAR-10, ImageNet)
- Evaluation: Accuracy, F1-score, computational efficiency

## Results
- 15% improvement over baselines
- State-of-the-art on 3 benchmarks
- Computational savings of 40%

## Generated by
- Model: gemma3:4b
- Date: 2025-12-02
- Tokens: 1,247 input, 387 output

Skip Existing Summaries: The summarization workflow automatically detects and skips generation for papers that already have summary files. When running scripts/literature_search.py:

1. File existence check - Before generating a summary, the workflow checks if data/summaries/{citation_key}_summary.md already exists
2. Automatic skip - If the file exists, summarization is skipped and the existing file is used
3. Progress tracking - Skipped summaries are still tracked in progress and marked as "summarized"
4. Idempotent runs - Multiple executions with the same papers won't regenerate summaries unnecessarily

This ensures efficient processing and prevents duplicate work when resuming interrupted runs or re-running the same search.

Workflow Integration

Literature Search Pipeline

1. Search - Use infrastructure/literature/ CLI or API
2. Download - Automatic PDF retrieval with fallback to open access
3. Index - Add to library.json and references.bib
4. Summarize - Optional LLM summarization to summaries/ (automatically skips existing summaries)
5. Cite - Reference in manuscript with \cite{key}

Standalone Operation

Note: This bibliography is separate from any manuscript system:

• Maintained independently in this repository
• Can be manually copied to manuscript systems if needed
• No automatic synchronization with external systems

Usage Examples

Search and Download

# Search arXiv and Semantic Scholar
python3 -m infrastructure.literature.core.cli search "machine learning transformers" --limit 10

# Search and download PDFs
python3 -m infrastructure.literature.core.cli search "quantum computing" --download

# Use specific sources
python3 -m infrastructure.literature.core.cli search "neural networks" --sources arxiv,semanticscholar

Interactive Summarization

# Interactive search and summarize with LLM
./run_literature.sh                # Interactive menu (select search or summarize)
python3 scripts/literature_search.py --search     # Search for papers
python3 scripts/literature_search.py --summarize  # Generate summaries
# Or directly:
python3 scripts/literature_search.py --search     # Search for papers
python3 scripts/literature_search.py --summarize  # Generate summaries

Workflow:

1. Prompts for comma-separated keywords
2. Searches across sources (union of results)
3. Downloads PDFs to data/pdfs/
4. Generates summaries to data/summaries/
5. Updates references.bib and library.json

Library Management

# View library statistics
python3 -m infrastructure.literature.core.cli library stats

# List all papers
python3 -m infrastructure.literature.core.cli library list

# Export library as JSON
python3 -m infrastructure.literature.core.cli library export --output my_library.json

Data Management

Library Statistics (verified 2026-08-29)

library.json currently holds 456 entries ("count": 456, last updated 2025-12-16T05:10:15Z). Per-domain counts are not tracked by the library schema; recompute from entries[*].source if needed rather than trusting older prose claims.

Quality Metrics (unverified — recompute from library.json before citing)

• Metadata/DOI/PDF coverage percentages below are legacy prose claims, not recomputed from the current library.json:
• Citation keys: All unique and BibTeX-compatible
• Source diversity: Multiple academic databases represented

Maintenance

Regular cleanup:

# Remove papers without PDFs (space management)
python3 -m infrastructure.literature.core.cli library cleanup --no-pdf

# Check library statistics
python3 -m infrastructure.literature.core.cli library stats

# Update metadata from sources
python3 -m infrastructure.literature.core.cli library refresh

Integration with Research Workflow

Paper Discovery Phase

1. Keyword search across multiple sources
2. Rapid PDF download with Unpaywall fallback
3. Quick title/abstract screening from library.json
4. LLM summarization for detailed review

Manuscript Writing Phase

1. Citation insertion using \cite{key} in markdown
2. Automatic bibliography generation
3. Cross-reference validation during PDF build
4. Reference checking in final manuscript

Publication Phase

1. Bibliography export for submission systems
2. DOI verification for all cited works
3. Open access tracking for compliance
4. Citation network analysis (future feature)

File Formats and Standards

JSON Schema (library.json)

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "version": {"type": "string"},
    "updated": {"type": "string", "format": "date-time"},
    "count": {"type": "integer"},
    "entries": {
      "type": "object",
      "patternProperties": {
        ".*": {
          "type": "object",
          "properties": {
            "citation_key": {"type": "string"},
            "title": {"type": "string"},
            "authors": {"type": "array", "items": {"type": "string"}},
            "year": {"type": "integer"},
            "doi": {"type": "string"},
            "source": {"type": "string"},
            "url": {"type": "string"},
            "pdf_path": {"type": "string"},
            "added_date": {"type": "string", "format": "date-time"},
            "abstract": {"type": "string"},
            "venue": {"type": "string"},
            "citation_count": {"type": "integer"}
          },
          "required": ["citation_key", "title", "authors", "year", "source", "url", "added_date", "abstract"]
        }
      }
    }
  }
}

BibTeX Standards

• Citation keys: Lowercase, descriptive, unique
• Author format: "Last, First" or "First Last" (BibTeX handles conversion)
• DOI inclusion: When available, preferred over URLs
• Journal formatting: Standard abbreviations when applicable

Backup and Recovery

Backup Strategy

# Backup
tar -czf data_backup_$(date +%Y%m%d).tar.gz data/

# Selective backup (metadata only)
cp data/library.json data/references.bib backup/

# Restore from backup
tar -xzf literature_backup_20251202.tar.gz

Data Integrity

Validation checks:

• JSON schema compliance
• BibTeX syntax validation
• File path existence verification
• Citation key uniqueness
• DOI format validation

Performance Considerations

Storage Requirements

• Average PDF size: 2.3 MB
• Total storage: ~1.1 GB for hundreds of PDFs
• Metadata overhead: ~15 MB for JSON/BibTeX
• Growth rate: ~50-100 papers/month

Search Performance

• Local search: Instant (JSON queries)
• Source APIs: Rate-limited (3-5 seconds between requests)
• Download speed: Network dependent (typically 1-10 MB/s)
• LLM summarization: 2-5 minutes per paper (depending on model)

Troubleshooting

Common Issues

PDF Download Failures:

# Check failed downloads log
cat data/failed_downloads.json

# Retry specific paper
python3 -m infrastructure.literature.core.cli download smith2024machine

# Enable Unpaywall fallback
export LITERATURE_USE_UNPAYWALL=true
export UNPAYWALL_EMAIL=your@email.com

Citation Errors:

# Check library statistics
python3 -m infrastructure.literature.core.cli library stats

# List all papers to verify entries
python3 -m infrastructure.literature.core.cli library list

Library Corruption:

# Backup current state
cp data/library.json data/library.json.backup

# Rebuild from BibTeX (if needed)
python3 -m infrastructure.literature.core.cli library rebuild

Future Enhancements

Planned Features

• Citation network visualization - Graph of paper relationships
• Topic modeling - Automatic categorization of papers
• Collaborator discovery - Find co-authors and related researchers
• Reading list management - Curated collections and annotations
• Integration with reference managers - Zotero, Mendeley sync
• Automated literature reviews - LLM-generated review articles

See Also

• ../infrastructure/literature/AGENTS.md - Literature search implementation
• ../infrastructure/literature/README.md - CLI usage guide
• ../infrastructure/llm/AGENTS.md - LLM summarization details
• ../scripts/literature_search.py - Literature search thin orchestrator