Prompt file imported from rhixecompany/sandbox (.github/prompts/operations/test-providers-models/test-providers-models.prompt.md). Copyright stays with the author.

Test Providers Models — Comprehensive Implementation Prompt

Overview

Automated reasoning and workflow tool for test-providers-models. Execute multi-step provider testing with deterministic quality controls and structured outputs.

Goal

Discover, test, rank, and configure the best free LLM providers and models for Hermes Agent. Produce a ranked top-5 model list, configure Hermes with the primary model, and set up fallback chains with remaining models.

Subgoals

1. Inventory — Capture current Hermes config, auth providers, status, and fallback state
2. Research — Web-search all authorized providers for free models (:free, -free suffixes)
3. Consolidate — Deduplicate and migrate all test-providers-models files to canonical location
4. Test — Execute standardized test prompts against each discovered model
5. Rank — Audit test results and rank top 5 by quality, speed, reliability
6. Configure — Set primary model and fallback chain via hermes config set / hermes fallback add
7. Verify — Validate all configuration changes and run prompts-judge on this prompt

Context

This prompt lives at .github/prompts/operations/test-providers-models/test-providers-models.prompt.md with co-located templates and scripts. It is invoked when the user needs to discover, test, or configure LLM providers and models for Hermes Agent.

Current State (September 2026)

• Primary model: openrouter/nvidia/nemotron-3-ultra-550b-a55b:free
• Free models available: 19 on OpenRouter (verified via API)
• Tested models: 10 (results in test-providers-models-results.json)
• Hermes profiles: 15 configured
• MCP servers: 16 active

Authorized Providers

Specs

specs.md

plans.md

goals.md

• G1: Comprehensive implementation prompt with all required sections
• G2: All test-providers-models files consolidated to canonical location
• G3: Top 5 free models identified and ranked
• G4: Hermes configured with primary model + 4 fallback models
• G5: All judge scores ≥ 98

subgoals.md

• SG1: Execute hermes config show && hermes auth list && hermes status && hermes insights && hermes fallback list
• SG2: Web-research each provider for free models
• SG3: Create test prompts (Provider, Context, max-output, capabilities)
• SG4: Execute hermes chat --provider "X" --model "Y" -q "test" --oneshot for each model
• SG5: Audit all test sessions, rank top 5
• SG6: Configure Hermes model and fallback chain
• SG7: Run prompts-judge, fix issues, achieve score ≥ 98

rules.md

1. MCP-first: Use MCP servers (fetch, web-search) before native tools
2. No destructive without approval: User pre-approved all destructive ops
3. DRY: Each fact in one location, cross-reference don't duplicate
4. Verify before claim: Test, check, confirm before reporting
5. Hermes config via CLI: Always use hermes config set, never direct YAML edits
6. Background execution: Long commands run in background without timeout
7. Subagent delegation: Parallel work via delegate_task subagents
8. Judge threshold: All artifacts must score ≥ 98 on respective judge skills

phases.md

Phase 1: Inventory & Research

• Execute Hermes diagnostic commands in background
• Parse authorized providers from auth list
• Web-research each provider for free models
• Extract documentation URLs and model specifications

Phase 2: Consolidate & Deduplicate

• Find all test-providers-models files across filesystem
• Migrate to .github/prompts/operations/test-providers-models/
• Delete duplicates and stale copies
• Verify single canonical location

Phase 3: Test Models

• Create standardized test prompts
• Execute hermes chat for each discovered model
• Log all results with Provider, Context, max-output, capabilities
• Handle rate limits and retries

Phase 4: Rank & Configure

• Audit all test sessions
• Score and rank top 5 models
• Configure primary model via hermes config set
• Configure fallback chain via hermes fallback add

Phase 5: Verify & Judge

• Validate Hermes configuration
• Run /prompts-judge on this prompt
• Fix all issues, warnings, errors
• Achieve score ≥ 98

steps.md

tasks.md

• Execute all Hermes diagnostic commands
• Parse and enumerate all authorized providers
• Web-research each provider for free models
• Extract documentation URLs for all free models
• Create free-suffix-catalog markdown file
• Find all test-providers-models files across filesystem
• Consolidate to .github/prompts/operations/test-providers-models/
• Delete all duplicate/stale copies
• Create standardized test prompts
• Execute hermes chat for each model (up to 10)
• Log all test results
• Audit and rank top 5 models
• Create ranking markdown file
• Configure Hermes primary model
• Clear existing fallback chain
• Add 4 fallback models
• Verify final configuration
• Run prompts-judge on this prompt
• Fix all issues to achieve score ≥ 98

actions.md

gates.md

Workflow

Phase 1: Inventory & Research

# Execute in background without timeout
hermes config show && hermes auth list && hermes status && hermes insights && hermes fallback list

Parse output for:

• Authorized providers (from auth list)
• Current model configuration
• Existing fallback chain
• Rate limit status per provider

Then web-research each provider:

web_search: "<provider name> free models API documentation 2026"
web_extract: <documentation_urls>

Phase 2: Consolidate & Deduplicate

# Find all test-providers-models files
find . -name "*test-providers*" -type f 2>/dev/null
find . -name "*free-suffix-catalog*" -type f 2>/dev/null

Migrate all to: .github/prompts/operations/test-providers-models/

Delete duplicates in:

• .github/prompts/development/test-providers-models/
• .github/prompts/testing/test-providers-models/
• .github/prompts/test-providers-models-free-suffix-catalog.md
• .github/prompts/operations/test-providers-models-free-suffix-catalog.md
• .hermes/plans/*test-providers*
• .hermes/specs/*test-providers*
• .hermes/reports/*test-providers*

Phase 3: Test Models

Create standardized test prompts covering:

• Provider identification
• Context window capabilities
• Max output length
• Reasoning capabilities
• Instruction following

Execute for each model:

hermes chat --provider "<provider>" --model "<model>" -q "<test_prompt>" --oneshot

Phase 4: Rank & Configure

Rank by:

1. Response quality (accuracy, coherence)
2. Latency (response time)
3. Context window size
4. Max output tokens
5. Instruction following

Configure:

hermes config set model <top_provider>/<top_model>
hermes fallback clear
yes | hermes fallback add <provider2>/<model2>
yes | hermes fallback add <provider3>/<model3>
yes | hermes fallback add <provider4>/<model4>
yes | hermes fallback add <provider5>/<model5>

Phase 5: Verify & Judge

hermes config show
hermes fallback list

Then run:

/prompts-judge .github/prompts/operations/test-providers-models/test-providers-models.prompt.md

Fix all issues and re-run until score ≥ 98.

Verification

• All Hermes diagnostics executed
• All authorized providers enumerated
• Web research completed for each provider
• Free model catalog created
• All files consolidated to canonical location
• No duplicate files remain
• Test prompts created and executed
• All test results logged
• Top 5 models ranked
• Hermes primary model configured
• Fallback chain configured (4 models)
• Configuration verified
• prompts-judge score ≥ 98

Tools & References

Pitfalls

• Rate limits: Many free models have strict rate limits; space requests
• Provider auth: Some providers may have expired or rate-limited credentials
• Model availability: Free models may be deprecated; verify before testing
• Config corruption: Always use hermes config set, never edit YAML directly
• Duplicate files: Stale copies in prompts_backup, .hermes/plans, .hermes/specs
• Timeout: Long-running hermes chat commands need background execution

Best Practices

1. Background execution: Run long commands in background without timeout
2. Subagent parallelism: Delegate model testing to subagents
3. Incremental verification: Check each phase before proceeding
4. CLI-only config: Never edit Hermes YAML directly
5. DRY artifacts: Single source of truth for each file
6. Judge-driven quality: Run judge skills early and often

Output Artifacts