How long does the integration take?

Change one base URL in an OpenAI-compatible client, or install the Python or TypeScript SDK; the quickstart routes a first request in a few minutes.

Will this lock us into another vendor?

No. The router is Apache-2.0 open source, your provider keys stay yours, and the API is OpenAI-compatible, so leaving is a change of base URL.

What happens when a provider fails?

The router falls back to the next model your policy allows and records the fallback in the trace, so an outage costs a retry, not an incident.

define-requirements - Prompt

Name: define-requirements
Author: BrandonT310442

Prompt file imported from BrandonT310442/AI-Judge-Take-Home-Assesment (.claude/commands/define-requirements.md). Fill in {{FEATURE}} before use. Copyright stays with the author.

Define Requirements - AI Judge Feature Specification

Creates a requirements document for AI Judge features based on the Besimple AI coding challenge brief. Maps out what needs to be accomplished for implementing evaluation features, judge management, and results viewing.

Usage

define-requirements - Creates requirements based on the coding challenge brief
define-requirements {{FEATURE}} - Define requirements for specific feature (data-ingestion, judge-management, evaluation-runner, results-view)

Process

1. Gather Context

Core Challenge Requirements:

React 18 + TypeScript + Vite application
Real LLM provider integration (OpenAI, Anthropic, or Gemini)
Backend persistence (Firebase, Supabase, or SQLite)
JSON data ingestion from sample_input.json format
Judge CRUD operations with persistence
Evaluation execution and storage
Results viewing with filters and statistics

Feature Areas:

Data Ingestion - JSON upload and parsing
Judge Management - CRUD for AI judges
Judge Assignment - Assign judges to questions
Evaluation Runner - Execute LLM evaluations
Results View - Display and filter results

2. Create Requirements Document

Output Directory: docs/requirements/ Output File: REQUIREMENTS_{FEATURE_NAME}.md

3. Requirements Document Structure

# Requirements: {Feature Name}

## Feature Summary
**Type**: {Data Management|AI Integration|UI Component|Backend Service}
**Priority**: {Core|Essential|Enhancement}
**Dependencies**: {Other features this depends on}

## Objective
{Clear statement of what this feature accomplishes for the AI Judge system}

## Functional Requirements

### Core Requirements
- **FR1**: Accept JSON file upload matching sample_input.json schema
- **FR2**: Parse and validate submission structure
- **FR3**: Persist data to backend (not localStorage)
- **FR4**: Handle multiple queues and submissions

### User Interface Requirements
- **UI1**: Drag-and-drop file upload interface
- **UI2**: Upload progress indication
- **UI3**: Validation error display
- **UI4**: Success confirmation

### Data Requirements
- **DR1**: Support submission schema with id, queueId, questions, answers
- **DR2**: Handle nested question/answer structures
- **DR3**: Maintain data integrity across features

## Technical Requirements

### Backend Requirements
- **TR1**: Use Firebase/Supabase/SQLite for persistence
- **TR2**: Implement proper error handling
- **TR3**: Support concurrent operations
- **TR4**: Maintain ACID properties for data operations

### Frontend Requirements
- **TR5**: React 18 with TypeScript
- **TR6**: Vite build configuration
- **TR7**: Responsive design
- **TR8**: Loading and error states

### Integration Requirements
- **IR1**: LLM Provider API integration (OpenAI/Anthropic/Gemini)
- **IR2**: Secure API key management
- **IR3**: Rate limiting and retry logic
- **IR4**: Response parsing and error handling

## Acceptance Criteria

### Data Ingestion
- [ ] JSON file can be uploaded via UI
- [ ] File is validated against expected schema
- [ ] Data is persisted to backend storage
- [ ] User receives confirmation of successful upload
- [ ] Invalid files show clear error messages

### Judge Management
- [ ] Judges can be created with name, prompt, model, active flag
- [ ] Judges can be edited and updated
- [ ] Judges can be deactivated (soft delete)
- [ ] Judge list persists across page reloads
- [ ] Changes are immediately reflected in UI

### Judge Assignment
- [ ] UI shows questions grouped by queue
- [ ] Multiple judges can be assigned per question
- [ ] Assignments are persisted to backend
- [ ] Clear visual indication of assigned judges

### Evaluation Execution
- [ ] "Run AI Judges" triggers evaluation process
- [ ] Progress indication during execution
- [ ] LLM API calls include judge prompt + question + answer
- [ ] Verdicts (pass/fail/inconclusive) are captured
- [ ] Reasoning text is stored
- [ ] Error handling for API failures

### Results Display
- [ ] Results show submission, question, judge, verdict, reasoning
- [ ] Multi-select filters for judges and questions
- [ ] Verdict filter (pass/fail/inconclusive)
- [ ] Aggregate pass rate calculation
- [ ] Results persist in backend

## Out of Scope
- User authentication/authorization
- Multi-tenant support
- Real-time collaboration
- Export functionality (unless implemented as bonus)
- Historical version tracking

## Testing Requirements
- Unit tests for data parsing logic
- Integration tests for API calls
- UI component testing
- End-to-end workflow testing
- Error scenario coverage

## Performance Requirements
- Initial page load < 3 seconds
- File upload processing < 5 seconds for typical files
- LLM API timeout handling (30 second timeout)
- Smooth UI interactions (no blocking operations)

## Risk Assessment
- **API Rate Limits**: Implement queuing and retry logic
- **Large Files**: Set reasonable file size limits (10MB)
- **LLM Costs**: Provide usage estimates before execution
- **Data Loss**: Implement proper persistence and backup

## Success Metrics
- All functional requirements implemented
- Clean, maintainable code structure
- Comprehensive error handling
- Intuitive user experience
- Meeting evaluation rubric criteria

4. Feature-Specific Requirements

For Data Ingestion:

File format validation
Schema compatibility checking
Duplicate detection
Batch processing capability

For Judge Management:

Form validation rules
Prompt template suggestions
Model selection options
Soft delete implementation

For Evaluation Runner:

Queue management
Progress tracking
Partial failure handling
Result aggregation

For Results View:

Pagination for large datasets
Export capabilities (bonus)
Chart visualizations (bonus)
Real-time updates

Key Principles

User-Centric: Focus on smooth, intuitive workflows
Robust: Handle errors gracefully at every step
Performant: Optimize for responsive interactions
Maintainable: Clean code structure and documentation
Testable: Comprehensive test coverage

Success Criteria

✅ Requirements align with coding challenge brief
✅ All evaluation rubric points addressed
✅ Technical constraints considered
✅ Clear acceptance criteria defined
✅ Ready for implementation planning

The goal is a clear, comprehensive requirements document that ensures all aspects of the AI Judge system are properly specified.

Install define-requirements and it becomes one more candidate the router can pick - when it fits.

A listing is a routing target with a manifest: what it does, which domains it covers, what it costs and who publishes it. Once installed it sits beside your own models and tools, is scored like any other candidate for each request, and shows up in the trace when it wins. Ratings come from workspaces that installed it, one per account.

# Define Requirements - AI Judge Feature Specification Creates a requirements document for AI Judge features based on the Besimple AI coding challenge brief. Maps out what needs to be accomplished for implementing evaluation features, judge management, and results viewing. # Usage - `define-requirements` - Creates requirements based on the coding challenge brief - `define-requirements {{FEATURE}}` - Define requirements for specific feature (data-ingestion, judge-management, evaluation-runner, results-view) # Process ## 1. Gather Context **Core Challenge Requirements:** - React 18 + TypeScript + Vite application - Real LLM provider integration (OpenAI, Anthropic, or Gemini) - Backend persistence (Firebase, Supabase, or SQLite) - JSON data ingestion from sample_input.json format - Judge CRUD operations with persistence - Evaluation execution and storage - Results viewing with filters and statistics **Feature Areas:** 1. **Data Ingestion** - JSON upload and parsing 2. **Judge Management** - CRUD for AI judges 3. **Judge Assignment** - Assign judges to questions 4. **Evaluation Runner** - Execute LLM evaluations 5. **Results View** - Display and filter results ## 2. Create Requirements Document **Output Directory:** `docs/requirements/` **Output File:** `REQUIREMENTS_{FEATURE_NAME}.md` ## 3. Requirements Document Structure ```markdown # Requirements: {Feature Name} ## Feature Summary **Type**: {Data Management|AI Integration|UI Component|Backend Service} **Priority**: {Core|Essential|Enhancement} **Dependencies**: {Other features this depends on} ## Objective {Clear statement of what this feature accomplishes for the AI Judge system} ## Functional Requirements ### Core Requirements - **FR1**: Accept JSON file upload matching sample_input.json schema - **FR2**: Parse and validate submission structure - **FR3**: Persist data to backend (not localStorage) - **FR4**: Handle multiple queues and submissions ### User Interface Requirements - **UI1**: Drag-and-drop file upload interface - **UI2**: Upload progress indication - **UI3**: Validation error display - **UI4**: Success confirmation ### Data Requirements - **DR1**: Support submission schema with id, queueId, questions, answers - **DR2**: Handle nested question/answer structures - **DR3**: Maintain data integrity across features ## Technical Requirements ### Backend Requirements - **TR1**: Use Firebase/Supabase/SQLite for persistence - **TR2**: Implement proper error handling - **TR3**: Support concurrent operations - **TR4**: Maintain ACID properties for data operations ### Frontend Requirements - **TR5**: React 18 with TypeScript - **TR6**: Vite build configuration - **TR7**: Responsive design - **TR8**: Loading and error states ### Integration Requirements - **IR1**: LLM Provider API integration (OpenAI/Anthropic/Gemini) - **IR2**: Secure API key management - **IR3**: Rate limiting and retry logic - **IR4**: Response parsing and error handling ## Acceptance Criteria ### Data Ingestion - [ ] JSON file can be uploaded via UI - [ ] File is validated against expected schema - [ ] Data is persisted to backend storage - [ ] User receives confirmation of successful upload - [ ] Invalid files show clear error messages ### Judge Management - [ ] Judges can be created with name, prompt, model, active flag - [ ] Judges can be edited and updated - [ ] Judges can be deactivated (soft delete) - [ ] Judge list persists across page reloads - [ ] Changes are immediately reflected in UI ### Judge Assignment - [ ] UI shows questions grouped by queue - [ ] Multiple judges can be assigned per question - [ ] Assignments are persisted to backend - [ ] Clear visual indication of assigned judges ### Evaluation Execution - [ ] "Run AI Judges" triggers evaluation process - [ ] Progress indication during execution - [ ] LLM API calls include judge prompt + question + answer - [ ] Verdicts (pass/fail/inconclusive) are captured - [ ] Reasoning text is stored - [ ] Error handling for API failures ### Results Display - [ ] Results show submission, question, judge, verdict, reasoning - [ ] Multi-select filters for judges and questions - [ ] Verdict filter (pass/fail/inconclusive) - [ ] Aggregate pass rate calculation - [ ] Results persist in backend ## Out of Scope - User authentication/authorization - Multi-tenant support - Real-time collaboration - Export functionality (unless implemented as bonus) - Historical version tracking ## Testing Requirements - Unit tests for data parsing logic - Integration tests for API calls - UI component testing - End-to-end workflow testing - Error scenario coverage ## Performance Requirements - Initial page load < 3 seconds - File upload processing < 5 seconds for typical files - LLM API timeout handling (30 second timeout) - Smooth UI interactions (no blocking operations) ## Risk Assessment - **API Rate Limits**: Implement queuing and retry logic - **Large Files**: Set reasonable file size limits (10MB) - **LLM Costs**: Provide usage estimates before execution - **Data Loss**: Implement proper persistence and backup ## Success Metrics - All functional requirements implemented - Clean, maintainable code structure - Comprehensive error handling - Intuitive user experience - Meeting evaluation rubric criteria ``` ## 4. Feature-Specific Requirements ### For Data Ingestion: - File format validation - Schema compatibility checking - Duplicate detection - Batch processing capability ### For Judge Management: - Form validation rules - Prompt template suggestions - Model selection options - Soft delete implementation ### For Evaluation Runner: - Queue management - Progress tracking - Partial failure handling - Result aggregation ### For Results View: - Pagination for large datasets - Export capabilities (bonus) - Chart visualizations (bonus) - Real-time updates ## Key Principles 1. **User-Centric**: Focus on smooth, intuitive workflows 2. **Robust**: Handle errors gracefully at every step 3. **Performant**: Optimize for responsive interactions 4. **Maintainable**: Clean code structure and documentation 5. **Testable**: Comprehensive test coverage ## Success Criteria - ✅ Requirements align with coding challenge brief - ✅ All evaluation rubric points addressed - ✅ Technical constraints considered - ✅ Clear acceptance criteria defined - ✅ Ready for implementation planning The goal is a clear, comprehensive requirements document that ensures all aspects of the AI Judge system are properly specified.

{ "prompt": "# Define Requirements - AI Judge Feature Specification\n\nCreates a requirements document for AI Judge features based on the Besimple AI coding challenge brief. Maps out what needs to be accomplished for implementing evaluation features, judge management, and results viewing.\n\n# Usage\n\n- `define-requirements` - Creates requirements based on the coding challenge brief\n- `define-requirements {{FEATURE}}` - Define requirements for specific feature (data-ingestion, judge-management, evaluation-runner, results-view)\n\n# Process\n\n## 1. Gather Context\n\n**Core Challenge Requirements:**\n- React 18 + TypeScript + Vite application\n- Real LLM provider integration (OpenAI, Anthropic, or Gemini)\n- Backend persistence (Firebase, Supabase, or SQLite)\n- JSON data ingestion from sample_input.json format\n- Judge CRUD operations with persistence\n- Evaluation execution and storage\n- Results viewing with filters and statistics\n\n**Feature Areas:**\n1. **Data Ingestion** - JSON upload and parsing\n2. **Judge Management** - CRUD for AI judges\n3. **Judge Assignment** - Assign judges to questions\n4. **Evaluation Runner** - Execute LLM evaluations\n5. **Results View** - Display and filter results\n\n## 2. Create Requirements Document\n\n**Output Directory:** `docs/requirements/`\n**Output File:** `REQUIREMENTS_{FEATURE_NAME}.md`\n\n## 3. Requirements Document Structure\n\n```markdown\n# Requirements: {Feature Name}\n\n## Feature Summary\n**Type**: {Data Management|AI Integration|UI Component|Backend Service}\n**Priority**: {Core|Essential|Enhancement}\n**Dependencies**: {Other features this depends on}\n\n## Objective\n{Clear statement of what this feature accomplishes for the AI Judge system}\n\n## Functional Requirements\n\n### Core Requirements\n- **FR1**: Accept JSON file upload matching sample_input.json schema\n- **FR2**: Parse and validate submission structure\n- **FR3**: Persist data to backend (not localStorage)\n- **FR4**: Handle multiple queues and submissions\n\n### User Interface Requirements\n- **UI1**: Drag-and-drop file upload interface\n- **UI2**: Upload progress indication\n- **UI3**: Validation error display\n- **UI4**: Success confirmation\n\n### Data Requirements\n- **DR1**: Support submission schema with id, queueId, questions, answers\n- **DR2**: Handle nested question/answer structures\n- **DR3**: Maintain data integrity across features\n\n## Technical Requirements\n\n### Backend Requirements\n- **TR1**: Use Firebase/Supabase/SQLite for persistence\n- **TR2**: Implement proper error handling\n- **TR3**: Support concurrent operations\n- **TR4**: Maintain ACID properties for data operations\n\n### Frontend Requirements\n- **TR5**: React 18 with TypeScript\n- **TR6**: Vite build configuration\n- **TR7**: Responsive design\n- **TR8**: Loading and error states\n\n### Integration Requirements\n- **IR1**: LLM Provider API integration (OpenAI/Anthropic/Gemini)\n- **IR2**: Secure API key management\n- **IR3**: Rate limiting and retry logic\n- **IR4**: Response parsing and error handling\n\n## Acceptance Criteria\n\n### Data Ingestion\n- [ ] JSON file can be uploaded via UI\n- [ ] File is validated against expected schema\n- [ ] Data is persisted to backend storage\n- [ ] User receives confirmation of successful upload\n- [ ] Invalid files show clear error messages\n\n### Judge Management\n- [ ] Judges can be created with name, prompt, model, active flag\n- [ ] Judges can be edited and updated\n- [ ] Judges can be deactivated (soft delete)\n- [ ] Judge list persists across page reloads\n- [ ] Changes are immediately reflected in UI\n\n### Judge Assignment\n- [ ] UI shows questions grouped by queue\n- [ ] Multiple judges can be assigned per question\n- [ ] Assignments are persisted to backend\n- [ ] Clear visual indication of assigned judges\n\n### Evaluation Execution\n- [ ] \"Run AI Judges\" triggers evaluation process\n- [ ] Progress indication during execution\n- [ ] LLM API calls include judge prompt + question + answer\n- [ ] Verdicts (pass/fail/inconclusive) are captured\n- [ ] Reasoning text is stored\n- [ ] Error handling for API failures\n\n### Results Display\n- [ ] Results show submission, question, judge, verdict, reasoning\n- [ ] Multi-select filters for judges and questions\n- [ ] Verdict filter (pass/fail/inconclusive)\n- [ ] Aggregate pass rate calculation\n- [ ] Results persist in backend\n\n## Out of Scope\n- User authentication/authorization\n- Multi-tenant support\n- Real-time collaboration\n- Export functionality (unless implemented as bonus)\n- Historical version tracking\n\n## Testing Requirements\n- Unit tests for data parsing logic\n- Integration tests for API calls\n- UI component testing\n- End-to-end workflow testing\n- Error scenario coverage\n\n## Performance Requirements\n- Initial page load < 3 seconds\n- File upload processing < 5 seconds for typical files\n- LLM API timeout handling (30 second timeout)\n- Smooth UI interactions (no blocking operations)\n\n## Risk Assessment\n- **API Rate Limits**: Implement queuing and retry logic\n- **Large Files**: Set reasonable file size limits (10MB)\n- **LLM Costs**: Provide usage estimates before execution\n- **Data Loss**: Implement proper persistence and backup\n\n## Success Metrics\n- All functional requirements implemented\n- Clean, maintainable code structure\n- Comprehensive error handling\n- Intuitive user experience\n- Meeting evaluation rubric criteria\n```\n\n## 4. Feature-Specific Requirements\n\n### For Data Ingestion:\n- File format validation\n- Schema compatibility checking\n- Duplicate detection\n- Batch processing capability\n\n### For Judge Management:\n- Form validation rules\n- Prompt template suggestions\n- Model selection options\n- Soft delete implementation\n\n### For Evaluation Runner:\n- Queue management\n- Progress tracking\n- Partial failure handling\n- Result aggregation\n\n### For Results View:\n- Pagination for large datasets\n- Export capabilities (bonus)\n- Chart visualizations (bonus)\n- Real-time updates\n\n## Key Principles\n\n1. **User-Centric**: Focus on smooth, intuitive workflows\n2. **Robust**: Handle errors gracefully at every step\n3. **Performant**: Optimize for responsive interactions\n4. **Maintainable**: Clean code structure and documentation\n5. **Testable**: Comprehensive test coverage\n\n## Success Criteria\n\n- ✅ Requirements align with coding challenge brief\n- ✅ All evaluation rubric points addressed\n- ✅ Technical constraints considered\n- ✅ Clear acceptance criteria defined\n- ✅ Ready for implementation planning\n\nThe goal is a clear, comprehensive requirements document that ensures all aspects of the AI Judge system are properly specified.", "variables": [ "FEATURE" ], "notes": "Creates a requirements document for AI Judge features based on the Besimple AI coding challenge brief. Maps out what needs to be accomplished for implementing evaluation features, judge management, and results viewing. Tools: Read, Write, Grep, Glob, Task.", "metadata": { "source": { "provider": "github-claude-commands", "repository": "https://github.com/BrandonT310442/AI-Judge-Take-Home-Assesment", "path": ".claude/commands/define-requirements.md", "ref": "db338d32532b54ceb00e9d43c65a640582c4bf5c", "url": "https://github.com/BrandonT310442/AI-Judge-Take-Home-Assesment/blob/db338d32532b54ceb00e9d43c65a640582c4bf5c/.claude/commands/define-requirements.md", "key": "BrandonT310442/AI-Judge-Take-Home-Assesment/.claude/commands/define-requirements.md" } } }

define-requirements

About

Define Requirements - AI Judge Feature Specification

Usage

Process

1. Gather Context

2. Create Requirements Document

3. Requirements Document Structure

4. Feature-Specific Requirements

For Data Ingestion:

For Judge Management:

For Evaluation Runner:

For Results View:

Key Principles

Success Criteria

Install define-requirements and it becomes one more candidate the router can pick - when it fits.

What it costs to adopt - in engineering time.

One base URL

No vendor lock-in

Failover built in

Use it

Manifest

Reviews