grepai-storage-gob

Imported from yoanbernabeu/grepai-skills (skills/storage/grepai-storage-gob/SKILL.md) via skills.sh. Install upstream with npx skills add yoanbernabeu/grepai-skills --skill grepai-storage-gob. Copyright stays with the author.

GrepAI Storage with GOB

This skill covers using GOB (Go Binary) as the storage backend for GrepAI, the default and simplest option.

When to Use This Skill

• Single developer projects
• Small to medium codebases
• Simple setup without external dependencies
• Local development environments

What is GOB Storage?

GOB is Go's native binary serialization format. GrepAI uses it to store:

• Vector embeddings
• File metadata
• Chunk information

Everything is stored in a single local file.

Advantages

Limitations

Configuration

Default Configuration

GOB is the default backend. Minimal config:

# .grepai/config.yaml
store:
  backend: gob

Explicit Configuration

store:
  backend: gob
  # Index stored in .grepai/index.gob (automatic)

Storage Location

GOB storage creates files in your project's .grepai/ directory:

.grepai/
├── config.yaml    # Configuration
├── index.gob      # Vector embeddings
└── symbols.gob    # Symbol index for trace

File Sizes

Approximate .grepai/index.gob sizes:

Operations

Creating the Index

# Initialize project
grepai init

# Start indexing (creates index.gob)
grepai watch

Checking Index Status

grepai status

# Output:
# Index: .grepai/index.gob
# Files: 245
# Chunks: 1,234
# Size: 12.5 MB
# Last updated: 2025-01-28 10:30:00

Backing Up the Index

# Simple file copy
cp .grepai/index.gob .grepai/index.gob.backup

Clearing the Index

# Delete and re-index
rm .grepai/index.gob
grepai watch

Moving to a New Machine

# Copy entire .grepai directory
cp -r .grepai /path/to/new/location/

# Note: Only works if using same embedding model

Performance Considerations

Memory Usage

GOB loads the entire index into RAM for searches:

Search Speed

GOB provides fast searches for typical codebases:

When to Upgrade

Consider PostgreSQL or Qdrant when:

• Index exceeds 1 GB
• Need concurrent access
• Want to share index across team
• Codebase has 50K+ files

.gitignore Configuration

Add .grepai/ to your .gitignore:

# GrepAI (machine-specific index)
.grepai/

Why: The index is machine-specific because:

• Contains binary embeddings
• Tied to the embedding model used
• Each machine should generate its own

Sharing Index (Not Recommended)

While you can copy the index file, it's not recommended because:

1. Must use identical embedding model
2. File paths are absolute
3. Different machines may have different code versions

Better approach: Each developer runs their own grepai watch.

Migrating to Other Backends

To PostgreSQL

1. Update config:

store:
  backend: postgres
  postgres:
    dsn: postgres://user:pass@localhost:5432/grepai

2. Re-index:

rm .grepai/index.gob
grepai watch

To Qdrant

1. Update config:

store:
  backend: qdrant
  qdrant:
    endpoint: localhost
    port: 6334

2. Re-index:

rm .grepai/index.gob
grepai watch

Common Issues

❌ Problem: Index file too large ✅ Solution: Add more ignore patterns or migrate to PostgreSQL/Qdrant

❌ Problem: Slow searches on large codebase ✅ Solution: Migrate to Qdrant for better performance

❌ Problem: Corrupted index ✅ Solution: Delete and re-index:

rm .grepai/index.gob .grepai/symbols.gob
grepai watch

❌ Problem: "Index not found" error ✅ Solution: Run grepai watch to create the index

Best Practices

1. Use for small/medium projects: Up to ~10K files
2. Add to .gitignore: Don't commit the index
3. Backup before major changes: Copy index.gob before experiments
4. Re-index after model changes: If you change embedding models
5. Monitor file size: Migrate if index exceeds 1GB

Output Format

GOB storage status:

✅ GOB Storage Configured

   Backend: GOB (local file)
   Index: .grepai/index.gob
   Size: 12.5 MB

   Contents:
   - Files: 245
   - Chunks: 1,234
   - Vectors: 1,234 × 768 dimensions

   Performance:
   - Search latency: <100ms
   - Memory usage: ~25 MB