recutils-js

Imported from chuchitrii/recutils-js (AGENTS.md). Install upstream with npx skills add chuchitrii/recutils-js. Copyright stays with the author.

AGENTS.md

Project Overview

This is a JavaScript/WebAssembly wrapper for the GNU recutils library. The project brings recutils functionality to JavaScript environments (Node.js, browsers, Obsidian plugins) by compiling the original C library to WebAssembly.

Approach

We don't reimplement recutils - we reuse the original.

• Primary approach: Compile original GNU recutils C code to WebAssembly
• Minimal custom C code: Only write new C when original code cannot compile to WASM
• Maximum compatibility: Leverage decades of GNU recutils development and testing
• Faithful behavior: Maintain identical functionality, argument parsing, and output formatting
• Strategic patching: Apply minimal patches only where WASM environment requires it

Current Status

✅ COMPLETED FEATURES:

• 7 GNU recutils utilities (recsel, recins, recdel, recset, recfmt, recinf, recfix)
• Full CLI compatibility with all GNU recutils options
• JavaScript API with multi-value field support
• WASM-based parse() method with native GNU recutils parsing
• Official test suite compatibility (336 of 342 tests passing)

Project Structure

Directory Layout

recutils-js/
├── dist/                   # Build output for npm package
│   ├── modules/           # Compiled WASM modules
│   │   ├── recsel.js     # WASM loader for recsel
│   │   ├── recsel.wasm   # Compiled recsel WebAssembly
│   │   └── ...           # Other utility modules
│   ├── recsel/           # recsel distribution files
│   │   ├── index.js      # ESM module entry
│   │   ├── index.cjs     # CommonJS entry
│   │   ├── cli.js        # CLI executable
│   │   └── wasm.js       # WASM wrapper
│   └── ...               # Other utilities follow same pattern
│
├── src/                    # Source code
│   ├── shared/           # Shared utilities
│   │   ├── recutils_core.c  # Core C functions
│   │   └── cli-utils.js     # CLI helper functions
│   ├── modules/          # Built WASM modules (gitignored)
│   ├── recsel/           # recsel implementation
│   │   ├── recutils_recsel.c  # C implementation
│   │   ├── index.js          # Three-tier API
│   │   ├── cli.js            # CLI entry point
│   │   └── wasm.js           # WASM wrapper
│   └── ...               # Other utilities follow same pattern
│
├── gnu-recutils/          # Git submodule - original GNU recutils
│   ├── src/              # GNU recutils C source
│   ├── torture/          # Official test suite
│   └── doc/              # Documentation
│
├── scripts/              # Build and test automation
│   ├── build-recutils.sh    # Build GNU recutils with Emscripten
│   ├── build.sh             # Build WASM modules
│   ├── build-dist.sh        # Prepare npm distribution
│   ├── test.sh              # Run official test suite
│   └── patches/             # Patches for WASM compatibility
│
├── tests/                # JavaScript test files
│   ├── recsel.test.js   # Unit tests for recsel
│   ├── integration.test.js  # Integration tests
│   ├── multiline.test.js    # Multiline field tests
│   └── ...
│
├── examples/             # Example applications
│   ├── exercise-notes/  # React app with recutils
│   └── cli-interactive/ # Interactive CLI interface
│
├── package.json         # NPM package configuration
├── README.md           # Main documentation
├── AGENTS.md           # This file - AI guidance
├── DEVELOPMENT.md      # Build instructions
├── UNIMPLEMENTED-FEATURES.md  # Known limitations
└── ROADMAP.md          # Future development plans

File Naming Conventions

• C Files: recutils_<utility>.c - WASM implementation
• JS Modules: index.js - Main API entry point
• CLI Scripts: cli.js - Command-line interface
• WASM Wrappers: wasm.js - Low-level WASM interface
• Tests: <utility>.test.js - Jest test files
• Scripts: <action>-<target>.sh - Shell scripts

Architecture

Architecture Layers

The project uses a modular architecture with separate folders for each utility. Each utility provides 3 consumption methods:

1. WASM Core Layer (src/shared/recutils_core.c + src/*/recutils_*.c)

   • Shared C core functions and utility-specific implementations
   • Direct wrappers around GNU recutils library functions
   • Functions: recutils_select_argv(), recutils_parse_json(), recutils_insert(), etc.
   • Memory management and error handling for WASM environment

2. WASM Wrapper Layer (src/*/wasm.js)

   • Thin JavaScript wrapper around each utility's WASM functions
   • Handles WASM memory allocation and argument marshaling
   • Direct interface to compiled utility modules

3. Three-Tier API Layer (src/*/index.js)

   • JS Options API: recsel(data, { expression: 'Age > "25"' })
   • String Args API: recselString(data, '-e "Age > 25" --count')
   • WASM Instance API: getWASMInstance() for direct access

4. CLI Layer (src/*/cli.js)

   • Command-line scripts: recsel-js, recins-js
   • 100% compatible with original GNU recutils CLI
   • Uses modular WASM builds for correct functionality

API Design

1. High-Level Functional API (recommended for most users)

   const { recsel, recins, parse } = require('recutils');
   const filtered = await recsel(data, { expression: 'Age > "25"' });
   const updated = await recins(data, { fields: { Name: 'John' } });
   const records = await parse(filtered);

2. String Arguments API (CLI-like but in JavaScript)

   const { recselString, recinsString } = require('recutils');
   const result = await recselString(data, '-e "Age > 25" --count');
   const updated = await recinsString(data, '-f Name -v John');

3. Direct Utility Import (modular access)

   const { recsel } = require('recutils/recsel');
   const { recins } = require('recutils/recins');
   const result = await recsel(data, { expression: 'Age > "25"' });

4. Command-Line Interface (full GNU recutils compatibility)

   recsel-js -e 'Age > "25"' -p Name,Age data.rec
   recins-js -f Name -v John -f Age -v 30 data.rec

Adding New Utilities

All GNU recutils utilities are already implemented. However, if a new utility were added to GNU recutils in the future, follow this systematic approach:

Step 1: Create Utility Folder

mkdir src/newutil

Step 2: Add C Implementation (src/newutil/recutils_newutil.c)

#include "../shared/recutils_core.h"

EMSCRIPTEN_KEEPALIVE
char* recutils_newutil(const char* input_data, char** argv, int argc) {
    // Use shared argument parsing logic
    parse_argv_with_original_logic(argc, argv);
    
    // Implement utility-specific logic
    // Follow the same pattern as other utilities
    
    return output;
}

Step 3: Add WASM Wrapper (src/newutil/wasm.js)

const NewutilModule = require('../../dist/modules/newutil.js');

class NewutilWASM {
    // Follow the same pattern as other WASM wrappers
    // Implement callWithArgv method
}

Step 4: Add Three-Tier API (src/newutil/index.js)

/**
 * Three-tier API for newutil
 */
async function newutil(inputData, options = {}) {
    // JS options object API
}

async function newutilString(inputData, argString = '') {
    // String arguments API
}

module.exports = { newutil, newutilString, getWASMInstance };

Step 5: Add CLI Script (src/newutil/cli.js)

#!/usr/bin/env node
// Follow the same pattern as other CLI scripts
// Handle all command-line arguments and options

Step 6: Update Package Configuration (package.json)

{
  "bin": {
    "recsel-js": "src/recsel/cli.js",
    "recins-js": "src/recins/cli.js", 
    "newutil-js": "src/newutil/cli.js"
  },
  "exports": {
    "./newutil": "./src/newutil/index.js"
  }
}

Step 7: Add Tests (scripts/test.sh)

# Copy test file
cp -r "$OLDPWD/gnu-recutils/torture/utils/newutil.sh" .

# Create wrapper script  
cat > newutil << 'EOF'
#!/bin/bash
exec node src/newutil/cli.js "$@"
EOF
chmod +x newutil

# Update test patterns to include newutil-*
grep -E "  (recsel|recins|newutil)-.*(ok|fail|error)" test_output.log

Development Conventions

Code Style

• Language: JavaScript with JSDoc annotations (no TypeScript)
• Comments: Avoid obvious comments - code should be self-documenting
  • ❌ Bad: // Increment counter before counter++
  • ❌ Bad: // Check if user exists before if (user)
  • ✅ Good: Comments explaining WHY, not WHAT
  • ✅ Good: Complex algorithm explanations
  • ✅ Good: Workarounds with context
• Testing: Test against original GNU recutils test suite for compatibility
• Automation: Always update scripts instead of manual fixes for reproducibility
• Build: Use npm scripts - all build logic is in scripts/ directory
• Git Strategy: Use git submodules for original recutils repository
• IMPORTANT: Always use npm run build, npm run test, etc. Never run scripts directly with ./scripts/

Testing Against Original GNU Recutils

When implementing or debugging functionality, always compare against the original GNU recutils:

# Install original GNU recutils for comparison testing
brew install recutils  # macOS
apt-get install recutils  # Ubuntu/Debian

# Compare behavior directly
echo "Name: Alice\nAge: 25" > test.rec

# Test original
recsel -e 'Age > "20"' test.rec
recins -e 'Name = "Alice"' -f Status -v Updated test.rec

# Test our implementation  
recsel-js -e 'Age > "20"' test.rec
recins-js -e 'Name = "Alice"' -f Status -v Updated test.rec

# Output should be identical

This approach helps ensure:

• Exact compatibility with GNU recutils behavior
• Correct argument parsing and option handling
• Proper output formatting and record structure
• Error handling that matches original utilities

Build System

See DEVELOPMENT.md for detailed build instructions, troubleshooting, and system requirements.

Key Commands for AI Development Work

• npm run build - Compile WASM modules after code changes (2-3 min)
• npm run test:official - Run against GNU recutils test suite (compatibility verification)
• npm run build-recutils - Full rebuild of GNU recutils libraries (20-25 min, rarely needed)

Build System Philosophy

• Automated patching: All compatibility fixes are automated in scripts
• Reproducible builds: Update scripts, never do manual fixes
• Incremental builds: Most changes only require npm run build
• GNU compatibility: Maintain exact compatibility with original recutils behavior

Testing Strategy

Two-Tier Testing Approach

Tier 1: CLI Compatibility Testing

• Method: Test our CLI utilities against the official GNU recutils test suite
• Command: npm run test:official
• Purpose: Ensure 100% behavioral compatibility with original GNU recutils
• Coverage: 336/342 testable tests passing (98.2% compatibility, 6 expected failures)
• Scope: All CLI options, argument parsing, output formatting, edge cases

Tier 2: JavaScript API Testing

• Method: Test JavaScript API functions against our own CLI implementations
• Command: npm run test:js
• Purpose: Verify that JS APIs produce identical results to CLI utilities
• Coverage: Comprehensive unit tests for all utilities and edge cases
• Scope: Options parsing, multi-value fields, error handling, multiline support

Testing Philosophy

1. CLI as Source of Truth: Our CLI utilities must match GNU recutils exactly
2. JS API Consistency: JavaScript functions must match our CLI behavior
3. Transitive Compatibility: If CLI matches GNU and JS matches CLI, then JS matches GNU
4. Comprehensive Coverage: Test both happy paths and error conditions
5. Cross-Implementation Verification: Same inputs produce identical outputs across all layers

Current Status

• GNU Compatibility: 98.2% (336/342 tests passing, 6 expected failures)
• JavaScript Tests: All major functionality covered including multiline fields, expressions, aggregation
• Expected Failures: 4 encryption tests, 1 external descriptor test, 1 template formatting test (see UNIMPLEMENTED-FEATURES.md)

Important Notes

DO (AI Development Guidelines):

• ✅ Test against official GNU recutils: Always compare behavior with original GNU recutils
• ✅ Update scripts for reproducibility: Automate fixes instead of manual changes
• ✅ Use WASM core for logic: Put complex processing in C, keep JavaScript thin
• ✅ Follow modular architecture: Each utility has separate C/JS/CLI layers
• ✅ Maintain GNU compatibility: API should match original recutils behavior exactly
• ✅ Use build commands properly: npm run build for most changes, avoid long rebuilds

DON'T (AI Development Anti-patterns):

• ❌ Manual fixes: Always update automation scripts instead
• ❌ Break GNU compatibility: Users expect identical behavior to original recutils
• ❌ Complex JavaScript logic: Move heavy processing to WASM core
• ❌ New documentation files: Consolidate into existing docs (README, DEVELOPMENT, AGENTS, UNIMPLEMENTED)
• ❌ Timeout on build-recutils.sh: This script can take 20-30 minutes legitimately

Current Priorities

1. Maintenance: Keep build system working and all tests passing
2. Documentation: Maintain comprehensive guides and examples
3. Performance: Continue leveraging WASM for optimal performance
4. Ecosystem: Support for various JavaScript environments (Node.js, browsers, Obsidian)
5. Stability: Focus on bug fixes and edge case handling over new features

Documentation Structure

• README.md: Project overview, installation, basic usage examples
• AGENTS.md (this file): AI assistant guidance, architecture, development patterns
• DEVELOPMENT.md: Build system, prerequisites, troubleshooting, contributing
• UNIMPLEMENTED-FEATURES.md: Known limitations and missing features
• ROADMAP.md: Future development plans and priorities

Useful References

• GNU recutils manual - Original recutils documentation
• Official test suite - Comprehensive compatibility tests
• Emscripten docs - WebAssembly compilation reference