Instruction file imported from qmx/qcode (.cursor/rules/vcr-testing-guide.mdc). Copyright stays with the author.

VCR Testing Guide for QCode

Description

Comprehensive VCR (Video Cassette Recorder) testing guide for QCode - covers recording/replaying HTTP interactions with Ollama API, modern VCR helper patterns, critical refactoring rules, E2E test setup, and deterministic test behavior. Essential for maintaining reliable integration tests with external LLM services while ensuring fast, reproducible test execution.

Overview

VCR (Video Cassette Recorder) testing allows us to record real HTTP interactions with external services (like Ollama) and replay them deterministically in tests. This provides the benefits of integration testing with real APIs while maintaining test reliability and speed.

Canonical Example

📋 See the complete working example: tests/integration/ollama-client.test.ts

This file demonstrates the modern VCR helper approach with clean, maintainable test code.

🚨 CRITICAL VCR REFACTORING RULES

NEVER break these rules when refactoring tests to use VCR:

1. ✅ Always Check for Existing Recordings First

Before claiming a refactoring is "complete", check tests/fixtures/recordings/ for required files.

2. ✅ Create Missing Recordings Immediately

Don't leave tests in a broken state. If recordings are missing, create them with:

NOCK_MODE=record npm test -- --testPathPattern="your-test-file.test.ts"

3. ✅ NEVER Manually Delete VCR Recordings

🚨 CRITICAL: Do NOT manually delete recording files with rm or similar commands. The VCR helper automatically overwrites existing recordings when in record mode. Manual deletion can break the recording/replay cycle and cause test failures.

❌ WRONG:

# DON'T DO THIS - breaks VCR workflow
rm tests/fixtures/recordings/*.json
rm tests/fixtures/recordings/file_read_query.json

✅ CORRECT:

# VCR helper automatically overwrites in record mode
NOCK_MODE=record npm test -- your-test-file.test.ts

4. ✅ Validate ALL Tests Pass Before Claiming Completion

Run the full test suite after refactoring. If any tests fail, fix them immediately.

5. ✅ Fix Test Assertion Issues Discovered During Recording

VCR recording may reveal that test expectations don't match actual LLM behavior. Fix assertions to match reality.

6. ✅ Ensure Tests Match Actual Implementation Behavior

No wishful thinking! If the engine only supports single function calls, don't test for multi-step workflows.

7. ✅ With VCR, Behavior MUST Be Deterministic

NEVER write comments like "LLM may choose differently" or "behavior may vary". VCR recordings are deterministic - same input = same output always.

8. ✅ Test Descriptions Must Be Honest

If a test only does single-step workflow, don't call it "complex multi-step workflow". Name tests accurately.

Example of BAD vs GOOD Test Refactoring:

❌ BAD (Dishonest, leaves broken tests):

it('should handle complex multi-step workflows', async () => {
  await vcr.withRecording('multi_step_workflow', async () => {
    const response = await engine.processQuery('List files then read package.json');
    
    // Wishful thinking - engine doesn't actually support this
    expect(response.toolsExecuted.length).toBeGreaterThan(1);
    // Note: LLM may choose differently - WRONG! VCR is deterministic
    expect(response.response).toContain('test-project');
  });
});

✅ GOOD (Honest, matches actual behavior):

it('should handle single-step workflow with complex query', async () => {
  await vcr.withRecording('multi_step_workflow', async () => {
    const response = await engine.processQuery('List files then read package.json');
    
    // Current engine implementation only supports single function calls
    // LLM interprets this as a file listing request (first part of query)
    expect(response.toolsExecuted.length).toBe(1);
    expect(response.toolsExecuted[0]).toBe('internal:files');
    expect(response.response).toContain('package.json');
  });
});

E2E Test VCR Patterns

Directory Structure for E2E Tests

tests/
├── e2e/                           # E2E tests with VCR
│   ├── function-calling.test.ts   # Engine + LLM function calling
│   ├── cli-*.test.ts             # CLI integration tests  
│   ├── workflow-*.test.ts        # Multi-step workflow tests
│   └── search-*.test.ts          # Search operation tests
├── fixtures/
│   └── recordings/               # VCR recordings for ALL tests
│       ├── function_calling/     # Organized by test suite
│       ├── cli/
│       ├── workflow/
│       └── search/
└── helpers/
    └── vcr-helper.ts            # Shared VCR helper

E2E Test Glob Patterns

Ensure Jest can find e2e tests by checking jest.config.js:

module.exports = {
  testMatch: [
    '**/tests/**/*.test.ts',      // Covers tests/unit/
    '**/tests/**/*.test.js',      // Covers tests/integration/
    '**/tests/e2e/**/*.test.ts',  // Explicitly include e2e/
  ],
  // ...
};

E2E VCR Test Template

import { QCodeEngine } from '../../src/core/engine.js';
import { OllamaClient } from '../../src/core/client.js';
import { ToolRegistry } from '../../src/core/registry.js';
import { FilesTool } from '../../src/tools/files.js';
import { WorkspaceSecurity } from '../../src/security/workspace.js';
import { getDefaultConfig } from '../../src/config/defaults.js';
import { setupVCRTests } from '../helpers/vcr-helper';

describe('Your E2E Test Suite', () => {
  let engine: QCodeEngine;
  const vcr = setupVCRTests(__filename);

  beforeEach(() => {
    // Setup real engine with all components
    const config = getDefaultConfig();
    const client = new OllamaClient(config.ollama);
    const workspaceSecurity = new WorkspaceSecurity(config.security);
    const toolRegistry = new ToolRegistry(config.security);
    const filesTool = new FilesTool(workspaceSecurity);

    toolRegistry.registerInternalTool(
      'files',
      filesTool.definition,
      filesTool.execute.bind(filesTool)
    );

    engine = new QCodeEngine(client, toolRegistry, config);
  });

  it('should do something specific and testable', async () => {
    await vcr.withRecording('descriptive_test_name', async () => {
      const response = await engine.processQuery('specific user query');

      // Test ACTUAL behavior, not aspirational behavior
      expect(response.complete).toBe(true);
      expect(response.toolsExecuted.length).toBe(1); // If engine only supports single calls
      
      vcr.recordingLog('✓ Response:', response.response);
    });
  });
});

VCR Infrastructure

Our VCR implementation uses:

• nock for HTTP recording/replaying
• VCR Helper Class (tests/helpers/vcr-helper.ts) - Encapsulates all complexity
• Automatic environment variable detection
• Fail-hard error handling with clear instructions

Environment Variables

• NOCK_MODE=record - Record new interactions with real APIs
• NOCK_MODE=replay (default) - Replay recorded interactions

Modern VCR Pattern (Recommended)

1. Test File Setup - Clean & Simple

import { OllamaClient } from '../../src/core/client';
import { getDefaultConfig } from '../../src/config/defaults';
import { setupVCRTests } from '../helpers/vcr-helper';

describe('OllamaClient VCR Tests', () => {
  let client: OllamaClient;
  const vcr = setupVCRTests(__filename);

  beforeEach(() => {
    const config = getDefaultConfig();
    client = new OllamaClient(config.ollama);
  });

  // Tests go here...
});

That's it! The VCR helper handles all the complexity:

• ✅ Automatic directory creation
• ✅ Environment variable detection
• ✅ Nock setup/teardown
• ✅ Recording lifecycle management

2. Individual Test Pattern - Super Clean

it('should handle some API interaction', async () => {
  await vcr.withRecording('descriptive_test_name', async () => {
    // Your actual test code - no VCR boilerplate needed!
    const result = await client.someAPICall();
    
    expect(result).toBeDefined();
    expect(result.someProperty).toBe('expectedValue');
    
    // Optional: Log important data during recording
    vcr.recordingLog('✓ Response received:', result);
  });
});

Benefits of the new pattern:

• 🚀 90% less boilerplate - Focus on test logic, not VCR mechanics
• 🛡️ Automatic error handling - Clear error messages if recordings missing
• 🔧 Environment encapsulation - No manual process.env.NOCK_MODE checks
• 📁 Auto-management - Recordings saved/loaded automatically

VCR Helper API

Core Methods

// Setup for entire test suite (use in describe blocks)
const vcr = setupVCRTests(__filename);

// Complete test workflow (recommended)
await vcr.withRecording('test_name', async () => {
  // Your test code here
});

// Manual control (advanced usage)
await vcr.loadRecording('test_name');    // Load recording for replay
await vcr.saveRecording('test_name');    // Save recording after test
vcr.recordingLog('Debug info:', data);   // Log only during recording

// Utilities
const exists = await vcr.hasRecording('test_name');
const recordings = await vcr.listRecordings();
await vcr.deleteRecording('test_name');

Environment Detection

// Check recording mode (encapsulated)
if (vcr.recording) {
  console.log('Recording new interactions');
} else {
  console.log('Replaying recorded interactions');
}

🔄 Iterative Development Workflow (Recommended)

💡 TIP: For test development, keep recording mode ON while iterating, then switch to replay mode when tests are stable.

Why Use Iterative Development?

When developing and debugging VCR tests, you want to:

• ✅ See real LLM responses while writing tests
• ✅ Iterate quickly without environment variable hassle
• ✅ Switch to deterministic replay when tests are working
• ✅ Avoid manual environment variable management

Method 1: Programmatic Recording Control (Easiest)

import { setupIterativeVCRTests } from '../helpers/vcr-helper';

describe('My New Feature Tests', () => {
  // Starts in recording mode for development
  const vcr = setupIterativeVCRTests(__filename);
  
  // Switch to replay mode when tests are stable (call this when ready)
  beforeAll(() => {
    // vcr.switchToReplay(); // Uncomment when tests are working
  });

  it('should handle new feature', async () => {
    await vcr.withRecording('new_feature_test', async () => {
      // Your test code - uses live Ollama while developing
      const response = await engine.processQuery('test this feature');
      
      expect(response.complete).toBe(true);
      vcr.recordingLog('✓ LLM Response:', response.response);
    });
  });
});

Method 2: forceRecord Option

import { setupVCRTests } from '../helpers/vcr-helper';

describe('My Feature Tests', () => {
  // Force recording mode regardless of environment variables
  const vcr = setupVCRTests(__filename, undefined, { forceRecord: true });
  
  it('should test feature', async () => {
    await vcr.withRecording('feature_test', async () => {
      // Always uses live Ollama calls
      const result = await client.complexAPICall();
      expect(result).toBeDefined();
    });
  });
});

Method 3: Manual Recording Control

import { setupVCRTests } from '../helpers/vcr-helper';

describe('Feature Tests', () => {
  const vcr = setupVCRTests(__filename);
  
  beforeAll(() => {
    // Enable recording for development iteration
    vcr.enableRecording();
    
    // Later: switch to replay mode when stable
    // vcr.disableRecording();
  });

  it('should test feature', async () => {
    await vcr.withRecording('test_name', async () => {
      // Test code here
    });
  });
});

Iterative Development Steps

Phase 1: Development (Recording Mode)

1. Start with recording enabled using setupIterativeVCRTests() or forceRecord: true
2. Make sure Ollama is running: ollama serve
3. Write your test logic and run tests repeatedly:

   npm test -- tests/your-test-file.test.ts --watch

4. Iterate freely - each test run uses live Ollama and updates recordings
5. Use vcr.recordingLog() to debug LLM responses during development

Phase 2: Stabilization (Switch to Replay)

1. When tests work correctly, switch to replay mode:

   beforeAll(() => {
     vcr.switchToReplay(); // or vcr.disableRecording()
   });

2. Run tests to ensure deterministic behavior:

   npm test -- tests/your-test-file.test.ts

3. Commit recordings to version control for CI/CD

Development Workflow Example

import { setupIterativeVCRTests } from '../helpers/vcr-helper';

describe('New Engine Feature', () => {
  const vcr = setupIterativeVCRTests(__filename);
  
  // PHASE 1: Development - keep this commented during iteration
  // PHASE 2: Stabilization - uncomment when tests are working
  // beforeAll(() => {
  //   vcr.switchToReplay();
  // });

  it('should process complex queries correctly', async () => {
    await vcr.withRecording('complex_query_test', async () => {
      const response = await engine.processQuery('analyze this complex scenario');
      
      // Start with basic assertions, refine as you see real responses
      expect(response.complete).toBe(true);
      expect(response.toolsExecuted.length).toBeGreaterThan(0);
      
      // Debug during development
      vcr.recordingLog('✓ Tools used:', response.toolsExecuted);
      vcr.recordingLog('✓ Response:', response.response);
      
      // Add more specific assertions based on actual LLM behavior
      expect(response.response).toContain('analysis');
    });
  });
});

Benefits of Iterative Approach

• 🚀 Faster development - No environment variable switching
• 🔍 Real feedback - See actual LLM responses while coding
• 🎯 Accurate tests - Write assertions based on real behavior
• 🔄 Easy switching - Programmatic control instead of manual env vars
• 📼 Automatic recording - Recordings update automatically during iteration

When to Switch to Replay Mode

Switch from recording to replay when:

• ✅ Your test logic is working correctly
• ✅ Assertions match actual LLM behavior
• ✅ You're satisfied with the recorded interactions
• ✅ Ready to commit stable, deterministic tests

🎯 Remember: The goal is fast iteration during development, then deterministic replay for CI/CD.

Recording New VCR Tests

💡 For development, use the Iterative Development Workflow above. This traditional approach is for one-off recording.

Step 1: Write the Test

Use the clean vcr.withRecording() pattern above.

Step 2: Record Real Interactions

# Make sure Ollama is running
ollama serve

# Run tests in record mode to capture real API interactions
NOCK_MODE=record npm test -- tests/your-test-file.test.ts

Step 3: Verify Recordings

Check that recordings were created in tests/fixtures/recordings/:

• Each test should have its own .json file
• Files should contain realistic HTTP request/response data
• Review recordings to ensure no sensitive data is captured

Step 4: Test Replay Mode

# Run tests in replay mode (default) to ensure deterministic behavior
npm test -- tests/your-test-file.test.ts

Best Practices

1. Naming Conventions

• Use descriptive snake_case test names: basic_chat, function_calling, list_models
• Test names become recording filenames, so make them clear and unique

2. Test Organization

describe('Model Management', () => {
  it('should list available models', async () => {
    await vcr.withRecording('list_models', async () => {
      // Test code...
    });
  });
});

describe('Chat Completion', () => {
  it('should handle basic chat', async () => {
    await vcr.withRecording('basic_chat', async () => {
      // Test code...
    });
  });
});

3. Recording Management

• Keep recordings in version control for deterministic CI/CD
• 🚨 NEVER manually delete recordings - VCR helper automatically overwrites files in record mode
• Re-record when API contracts change: NOCK_MODE=record npm test
• Clean up obsolete recordings when test names change by renaming tests, not deleting files
• The VCR helper handles all file lifecycle management automatically

Recording Lifecycle:

1. New recordings: VCR helper creates files automatically in record mode
2. Updating recordings: VCR helper overwrites existing files automatically
3. Obsolete recordings: Remove by changing test names, not by manual file deletion
4. Missing recordings: VCR helper fails with clear instructions to re-record

4. Error Handling - Automatic!

The VCR helper automatically:

• ❌ FAILS HARD if recordings are missing in replay mode
• 📝 Provides clear instructions for fixing missing recordings
• 🔍 Shows exact commands to re-record tests

Example error message:

❌ VCR Recording missing for test "basic_chat"
Recording file not found: /path/to/recordings/basic_chat.json

To fix this:
1. Run: NOCK_MODE=record npm test -- tests/integration/ollama-client.test.ts
2. Ensure Ollama is running: ollama serve
3. Check that recording was created in tests/fixtures/recordings/

5. Debugging During Recording

it('should handle complex interaction', async () => {
  await vcr.withRecording('complex_test', async () => {
    const result = await client.complexCall();
    
    // These logs only appear in record mode
    vcr.recordingLog('✓ API response:', result);
    vcr.recordingLog('✓ Model used:', result.model);
    
    expect(result).toBeDefined();
  });
});

File Organization

tests/
├── helpers/
│   └── vcr-helper.ts         # VCR helper class (reusable)
├── fixtures/
│   └── recordings/           # VCR recordings (JSON files)
│       ├── basic_chat.json
│       ├── function_calling.json
│       └── list_models.json
├── integration/              # VCR integration tests
│   ├── ollama-client.test.ts # Example VCR test file
│   └── engine.test.ts        # Engine VCR tests
├── e2e/                      # End-to-end VCR tests  
│   ├── function-calling.test.ts
│   ├── cli-*.test.ts
│   └── workflow-*.test.ts
└── unit/                     # Regular unit tests (no HTTP)

Migration from Old VCR Pattern

Before (Manual VCR - 50+ lines of boilerplate)

beforeEach(() => {
  if (process.env.NOCK_MODE === 'record') {
    nock.restore();
    nock.recorder.rec({ /* ... */ });
  } else {
    nock.disableNetConnect();
  }
});

it('should test something', async () => {
  const testName = 'test_name';
  
  if (process.env.NOCK_MODE !== 'record') {
    const recordingFile = path.join(recordingsPath, `${testName}.json`);
    try {
      const recordings = JSON.parse(await fs.readFile(recordingFile, 'utf-8'));
      // ... 15+ lines of nock setup ...
    } catch (error) {
      // ... 10+ lines of error handling ...
    }
  }
  
  const result = await someAPICall();
  expect(result).toBeDefined();
  
  if (process.env.NOCK_MODE === 'record') {
    const recordings = nock.recorder.play();
    // ... 10+ lines of recording save logic ...
  }
});

After (VCR Helper - 5 lines total!)

const vcr = setupVCRTests(__filename);

it('should test something', async () => {
  await vcr.withRecording('test_name', async () => {
    const result = await someAPICall();
    expect(result).toBeDefined();
  });
});

Common Issues - Now Solved!

✅ Recording Doesn't Work

• VCR helper provides clear error messages
• Automatic directory creation
• Built-in validation

✅ Replay Fails

• FAIL HARD with helpful instructions
• Automatic recording file discovery
• Clear commands to fix issues

✅ Flaky Tests

• Stable defaults built into VCR helper
• Consistent recording format
• No manual environment variable handling

Advanced Usage

Custom Recording Options

// Advanced VCR helper usage
const vcr = createVCRHelper(__filename, '../custom/recordings');

// Custom setup with options
beforeEach(() => {
  vcr.setupTest({ 
    enableRequestHeaders: true,
    outputObjects: true 
  });
});

Programmatic Recording Management

// Check if recording exists
if (await vcr.hasRecording('test_name')) {
  console.log('Recording available');
}

// List all recordings
const recordings = await vcr.listRecordings();
console.log('Available recordings:', recordings);

// Delete and re-record
await vcr.deleteRecording('old_test');

Example: Engine VCR Tests

import { QCodeEngine } from '../../src/core/engine';
import { setupVCRTests } from '../helpers/vcr-helper';

describe('QCodeEngine VCR Tests', () => {
  const vcr = setupVCRTests(__filename);
  let engine: QCodeEngine;

  beforeEach(() => {
    engine = new QCodeEngine(realOllamaClient, toolRegistry, config);
  });

  it('should process help query with real LLM', async () => {
    await vcr.withRecording('engine_help_query', async () => {
      const response = await engine.processQuery('help');
      
      expect(response.complete).toBe(true);
      expect(response.response).toContain('QCode AI Coding Assistant');
      
      vcr.recordingLog('✓ Engine response:', response.response);
    });
  });
});

This ensures our engine tests use real Ollama responses while remaining deterministic and fast in CI/CD.

Summary

The VCR helper provides a professional, maintainable testing infrastructure that:

• 🚀 Reduces boilerplate by 90%
• 🛡️ Provides automatic error handling
• 📁 Manages recordings automatically
• 🔧 Encapsulates environment complexity
• ✅ Ensures consistent, reliable tests

Use vcr.withRecording() for all new VCR tests - it's the modern, clean approach!

Remember: When refactoring tests to VCR, NEVER leave them broken. Create all recordings immediately and ensure 100% test pass rate before claiming completion.