Imported from rhixecompany/sandbox (.github/skills/provider-reliability-diagnostics/SKILL.md). Install upstream with npx skills add rhixecompany/sandbox --skill provider-reliability-diagnostics. Copyright stays with the author (MIT).

Provider Reliability Diagnostics

Overview

Automated reasoning and workflow tool for provider-reliability-diagnostics. Execute multi-step tasks with deterministic quality controls and structured outputs.

When to Use

• Provider connection failures in errors.log
• Fallback chain not activating as expected
• Auxiliary features (vision, compression, title gen) silently failing
• MCP server "Connection closed" at gateway startup
• Session abort mid-conversation with provider error
• After model/provider config changes

Verification Checklist

• Provider chain fully mapped (main → fallback → auxiliary)
• Each provider in chain passes health check
• Auxiliary providers configured explicitly (not auto when OpenRouter has no credits)
• MCP servers with transport mismatch disabled or fixed
• Gateway restart picks up all changes

The Provider Chain

Hermes attempts providers in this exact order when processing a request:

1. Main provider  ─── model.provider + model.default
   ↓ (fails: ConnectionError, 402, 429, timeout)
2. Fallback providers ─── fallback_providers[] (in order)
   ↓ (all fail)
3. Auxiliary aggregator ─── auxiliary.*.provider
   ↓ (fails: 402, auth error, or not configured)
4. Silent failure ─── session aborts or feature degrades

Each layer leaves distinct signatures in errors.log.

Phase 1: Quick Health Sweep

Run all three ACPX provider checks in parallel:

acpx copilot exec "Reply with exactly: COPILOT_ALIVE"
acpx qwen exec "Reply with exactly: QWEN_ALIVE"
opencode run "Reply with exactly: OPENCODE_ALIVE" --model opencode/big-pickle 2>&1 | tail -3

Healthy response: Each returns the exact string. Failure patterns:

• Copilot: "Authentication required" → re-auth GitHub token
• Qwen: "401" or timeout → check OpenRouter API key + credits
• OpenCode: cost-guard banner then valid response → cosmetic only

Phase 2: Log Trace — Identify the Broken Link

# 1. Pull WARNING/ERROR lines mentioning providers
grep -E "WARNING|ERROR" $LOCALAPPDATA/hermes/logs/errors.log \
  | grep -i "provider\|api_call\|retry\|credits\|auth" | tail -40

# 2. Classify the failure layer

Failure Signature Reference

Phase 3: Apply Fix at the Broken Link

Fix Main Provider Instability

If opencode-zen is flaky (intermittent connection errors, chunked read drops):

# Check if issue persists
curl -s -o /dev/null -w "%{http_code}" https://opencode.ai/zen/v1/models
# If endpoint is down, fallback should auto-activate.
# The fallback chain is configured in config.yaml:
#   fallback_providers:
#   - provider: copilot-acp
#     model: claude-sonnet-4.6

No fix needed on your side — the provider endpoint was temporarily unstable (Cloudflare-routed). Fallback handles it.

Fix Fallback Auth Failure

If copilot-acp says "Authentication required":

# Re-authenticate
gh auth status
copilot --version

Usually a transient GH token issue. Re-auth resolves it.

Fix Auxiliary 402 (OpenRouter Credits)

If auxiliary features fail with HTTP 402:

# 1. Check OpenRouter credit status
# Visit: https://openrouter.ai/settings/credits

# 2. Immediate workaround — set auxiliary to working provider
hermes config set auxiliary.vision.provider copilot-acp
hermes config set auxiliary.vision.model claude-sonnet-4.6
hermes config set auxiliary.compression.provider copilot-acp
hermes config set auxiliary.compression.model claude-sonnet-4.6
hermes config set auxiliary.title_generation.provider copilot-acp
hermes config set auxiliary.title_generation.model claude-sonnet-4.6

Fix MCP Transport Mismatch

If an MCP server uses a Docker/HTTP gateway command but Hermes connects via stdio:

# 1. Verify the MCP command type
# If it starts a long-running HTTP server, it's not stdio-compatible

# 2. Disable the misconfigured server
hermes config set mcp_servers.<name>.enabled false

# 3. Verify it's gone from startup
hermes mcp list 2>&1 | grep <name> || echo "✓ Removed from active MCP servers"

Phase 4: Verify Fix

# 1. Confirm config changes
grep -A2 "auxiliary" $LOCALAPPDATA/hermes/config.yaml

# 2. Re-run provider health check
acpx copilot exec "Reply with exactly: COPILOT_ALIVE"

# 3. Confirm MCP servers load cleanly (no Docker warnings)
tail -20 $LOCALAPPDATA/hermes/logs/gateway.log | grep "MCP"

# 4. Restart gateway to pick up all changes
hermes gateway run --replace

Pitfalls

α Provider works in VS Code but fails in Hermes

Hermes gateway loads credentials from .env + config.yaml. VS Code extensions use their own credential store. The two are independent — a provider working in VS Code does not mean it's configured for Hermes.

α Error chain is deep — don't fix the wrong layer

When you see "system failure: all providers exhausted" in a session, the actual root cause is often just one broken link. Trace back through the chain: was it the main provider? The fallback? The auxiliary? Fix that one link.

α "auto" auxiliary provider is not magic

auxiliary.*.provider: auto does NOT auto-detect a working provider. It tries the main provider first, then falls through to the OpenRouter aggregator pool. If OpenRouter has zero credits, everything after the main provider is dead.

α MCP "Connection closed" does not mean Docker is down

docker mcp gateway run starts an HTTP/SSE server. Hermes connects via stdio MCP protocol. The "connection closed" is a transport mismatch, not a Docker failure. Docker Desktop may be running fine.

Best Practices

1. Run provider health check weekly or after any config change
2. Set auxiliary providers explicitly if OpenRouter credits are limited
3. Keep fallback_providers configured even if main provider is stable — outages happen
4. After diagnosing a chain failure, add a reference file documenting the specific provider issue for faster triage next time
5. Restart gateway after any config change: hermes gateway run --replace

Related Skills

• hermes-agent-diagnostics-configuration — General Hermes health checks and config verification
• log-analysis-and-triage — General log file triage methodology
• acpx-agent-routing — ACPX agent selection by task type

Reference Files

• references/provider-chain-diagnostics.md — Session-specific diagnostic details, error transcripts, and reproduction recipes
• references/model-catalog-free-models.md — Model catalog endpoint, free model list (27 OpenRouter models), benchmark template
• scripts/test-free-models.py — Runnable benchmark script for free models via opencode-zen endpoint

Related Scripts (global)

~/AppData/Local/hermes/scripts/test_models.py — Multi-provider benchmark harness with --list-providers and --hermes flags. Covers all 6 configured providers.

Workflow

Phase 1: Preparation

Set up required environment, dependencies, and configuration for Provider Reliability Diagnostics — Chain Tracing, ACPX Validation, Auxiliary.

Phase 2: Execution

Run the primary Provider Reliability Diagnostics — Chain Tracing, ACPX Validation, Auxiliary operations according to the defined requirements.

Phase 3: Verification

Verify output, handle any errors, and confirm results meet expectations.

Phase 4: Completion

Document results, clean up resources, and finalize any deliverables.