How long does the integration take?

Change one base URL in an OpenAI-compatible client, or install the Python or TypeScript SDK; the quickstart routes a first request in a few minutes.

Will this lock us into another vendor?

No. The router is Apache-2.0 open source, your provider keys stay yours, and the API is OpenAI-compatible, so leaving is a change of base URL.

What happens when a provider fails?

The router falls back to the next model your policy allows and records the fallback in the trace, so an outage costs a retry, not an incident.

bench-experiment - Prompt - OpenSmartRoute

Name: bench-experiment
Author: alfredoperez

Prompt file imported from alfredoperez/speckit-companion (.claude/commands/bench-experiment.md). Fill in {{arguments}} before use. Copyright stays with the author.

Your task

Run one complete round of the benchmark and report what changed. The user says "run an experiment and bench it" and expects the whole thing without being asked for details.

The harness lives in the sibling speckit-bench repo and the app it measures in conduit. Its README.md is the reference; this command is the procedure.

1. Name the sweep

{{arguments}} is what the experiment is testing. Turn it into a short label — "hook anchors", "living specs off", "fresh baseline". If there is nothing to name, use today's date and call it a baseline repeat, which is also useful: it is how the noise floor gets measured.

2. Check the machine first

Twelve clones of a React app are about 8GB of files Spotlight has never seen, and indexing them has taken this machine to a load average of 260. Before baking:

ls ~/dev/projects/.metadata_never_index || echo "MISSING"

If it is missing, or if ps aux | grep mds_stores shows Spotlight busy, tell the user to run sudo mdutil -i off ~/dev/projects and wait for them. Do not start a round on a machine that is already struggling.

3. Bake

npm run compile                          # the driver dispatches the GUI preamble from dist/
node ../speckit-bench/sync-templates.mjs --sizes easy,medium,hard,oversized --ext code --sweep "<label>"

--ext code measures the extension in COMPANION_DIR, which is what an experiment on unreleased work needs. Use --ext latest only when the question is explicitly about the published build.

The bake prints the three versions it recorded and fails loudly if any cell can read what it is. Report the versions; do not report the arm mapping unless asked, and never pass it to a driver or a judge.

4. Prep and drive

node ../speckit-bench/run-all.mjs prep --sizes easy,medium,hard,oversized

Then one driver per cell, all twelve at once, following step 3 of /bench-run-all — the same GUI preamble, the same settle-wait, capture for the Companion arms only, and the cell's letter, never its arm. Twelve drivers in parallel is fine; the round costs the slowest cell rather than the sum.

Expect 20 to 30 minutes.

5. Measure — with nothing else running

node ../speckit-bench/run-all.mjs capture --sizes easy,medium,hard,oversized --no-reset

This runs a build and two test passes in each of twelve cells. Do not start the judges while it runs. Stacking twelve judge agents on top of it is what took the machine down; it costs nothing to wait the ten minutes.

--no-reset keeps the diffs so the judges have something to read.

6. Judge — after capture finishes, in small batches

A rubric judge per cell and a comparative reviewer per size, as /bench-capture steps 3 and 4 describe. Run them three or four at a time, not twelve. Then re-run capture once to fold the rubric into the composite.

If the machine is under load, or the user has other work to do, stop after step 5 and say so: the oracle, wall-clock and test columns are complete without the judges, and the composite can be filled in later.

7. Report

node ../speckit-bench/run-all.mjs compare --metric work
node ../speckit-bench/run-all.mjs compare --metric oracle
node ../speckit-bench/run-all.mjs compare --metric wall

Lead with what moved and whether it moved more than the spread of the sweeps before it. A change smaller than that spread has not been measured — say so plainly rather than reporting it as a result.

Read work, not overall, when comparing the stock arm to a Companion one. overall gives a quarter of its weight to lifecycle capture, which stock does not have and never will, so it caps stock at 75 by construction. work is correctness plus rubric and every arm can reach 100.

Then commit the round in speckit-bench — stats.jsonl, history.jsonl, runs/, reviews/, REPORT.md — with a message naming the sweep and what it was testing.

If something is incomplete

Say which cells are missing and which columns they cost. A round with eleven of twelve cells and no rubrics is still worth pushing; a round reported as complete when it is not is worse than no round.

Install bench-experiment and it becomes one more candidate the router can pick - when it fits.

A listing is a routing target with a manifest: what it does, which domains it covers, what it costs and who publishes it. Once installed it sits beside your own models and tools, is scored like any other candidate for each request, and shows up in the trace when it wins. Ratings come from workspaces that installed it, one per account.

## Your task Run one complete round of the benchmark and report what changed. The user says "run an experiment and bench it" and expects the whole thing without being asked for details. The harness lives in the sibling [`speckit-bench`](https://github.com/alfredoperez/speckit-bench) repo and the app it measures in [`conduit`](https://github.com/alfredoperez/conduit). Its `README.md` is the reference; this command is the procedure. ### 1. Name the sweep `{{arguments}}` is what the experiment is testing. Turn it into a short label — "hook anchors", "living specs off", "fresh baseline". If there is nothing to name, use today's date and call it a baseline repeat, which is also useful: it is how the noise floor gets measured. ### 2. Check the machine first Twelve clones of a React app are about 8GB of files Spotlight has never seen, and indexing them has taken this machine to a load average of 260. Before baking: ```bash ls ~/dev/projects/.metadata_never_index || echo "MISSING" ``` If it is missing, or if `ps aux | grep mds_stores` shows Spotlight busy, tell the user to run `sudo mdutil -i off ~/dev/projects` and wait for them. Do not start a round on a machine that is already struggling. ### 3. Bake ```bash npm run compile # the driver dispatches the GUI preamble from dist/ node ../speckit-bench/sync-templates.mjs --sizes easy,medium,hard,oversized --ext code --sweep "<label>" ``` `--ext code` measures the extension in `COMPANION_DIR`, which is what an experiment on unreleased work needs. Use `--ext latest` only when the question is explicitly about the published build. The bake prints the three versions it recorded and fails loudly if any cell can read what it is. Report the versions; do not report the arm mapping unless asked, and never pass it to a driver or a judge. ### 4. Prep and drive ```bash node ../speckit-bench/run-all.mjs prep --sizes easy,medium,hard,oversized ``` Then one driver per cell, all twelve at once, following step 3 of `/bench-run-all` — the same GUI preamble, the same settle-wait, capture for the Companion arms only, and **the cell's letter, never its arm**. Twelve drivers in parallel is fine; the round costs the slowest cell rather than the sum. Expect 20 to 30 minutes. ### 5. Measure — with nothing else running ```bash node ../speckit-bench/run-all.mjs capture --sizes easy,medium,hard,oversized --no-reset ``` This runs a build and two test passes in each of twelve cells. **Do not start the judges while it runs.** Stacking twelve judge agents on top of it is what took the machine down; it costs nothing to wait the ten minutes. `--no-reset` keeps the diffs so the judges have something to read. ### 6. Judge — after capture finishes, in small batches A rubric judge per cell and a comparative reviewer per size, as `/bench-capture` steps 3 and 4 describe. Run them **three or four at a time**, not twelve. Then re-run `capture` once to fold the rubric into the composite. If the machine is under load, or the user has other work to do, stop after step 5 and say so: the oracle, wall-clock and test columns are complete without the judges, and the composite can be filled in later. ### 7. Report ```bash node ../speckit-bench/run-all.mjs compare --metric work node ../speckit-bench/run-all.mjs compare --metric oracle node ../speckit-bench/run-all.mjs compare --metric wall ``` Lead with what moved and whether it moved more than the spread of the sweeps before it. A change smaller than that spread has not been measured — say so plainly rather than reporting it as a result. **Read `work`, not `overall`, when comparing the stock arm to a Companion one.** `overall` gives a quarter of its weight to lifecycle capture, which stock does not have and never will, so it caps stock at 75 by construction. `work` is correctness plus rubric and every arm can reach 100. Then commit the round in `speckit-bench` — `stats.jsonl`, `history.jsonl`, `runs/`, `reviews/`, `REPORT.md` — with a message naming the sweep and what it was testing. ### If something is incomplete Say which cells are missing and which columns they cost. A round with eleven of twelve cells and no rubrics is still worth pushing; a round reported as complete when it is not is worse than no round.

{ "prompt": "## Your task\n\nRun one complete round of the benchmark and report what changed. The user says \"run an experiment and bench it\" and expects the whole thing without being asked for details.\n\nThe harness lives in the sibling [`speckit-bench`](https://github.com/alfredoperez/speckit-bench) repo and the app it measures in [`conduit`](https://github.com/alfredoperez/conduit). Its `README.md` is the reference; this command is the procedure.\n\n### 1. Name the sweep\n\n`{{arguments}}` is what the experiment is testing. Turn it into a short label — \"hook anchors\", \"living specs off\", \"fresh baseline\". If there is nothing to name, use today's date and call it a baseline repeat, which is also useful: it is how the noise floor gets measured.\n\n### 2. Check the machine first\n\nTwelve clones of a React app are about 8GB of files Spotlight has never seen, and indexing them has taken this machine to a load average of 260. Before baking:\n\n```bash\nls ~/dev/projects/.metadata_never_index || echo \"MISSING\"\n```\n\nIf it is missing, or if `ps aux | grep mds_stores` shows Spotlight busy, tell the user to run `sudo mdutil -i off ~/dev/projects` and wait for them. Do not start a round on a machine that is already struggling.\n\n### 3. Bake\n\n```bash\nnpm run compile # the driver dispatches the GUI preamble from dist/\nnode ../speckit-bench/sync-templates.mjs --sizes easy,medium,hard,oversized --ext code --sweep \"<label>\"\n```\n\n`--ext code` measures the extension in `COMPANION_DIR`, which is what an experiment on unreleased work needs. Use `--ext latest` only when the question is explicitly about the published build.\n\nThe bake prints the three versions it recorded and fails loudly if any cell can read what it is. Report the versions; do not report the arm mapping unless asked, and never pass it to a driver or a judge.\n\n### 4. Prep and drive\n\n```bash\nnode ../speckit-bench/run-all.mjs prep --sizes easy,medium,hard,oversized\n```\n\nThen one driver per cell, all twelve at once, following step 3 of `/bench-run-all` — the same GUI preamble, the same settle-wait, capture for the Companion arms only, and **the cell's letter, never its arm**. Twelve drivers in parallel is fine; the round costs the slowest cell rather than the sum.\n\nExpect 20 to 30 minutes.\n\n### 5. Measure — with nothing else running\n\n```bash\nnode ../speckit-bench/run-all.mjs capture --sizes easy,medium,hard,oversized --no-reset\n```\n\nThis runs a build and two test passes in each of twelve cells. **Do not start the judges while it runs.** Stacking twelve judge agents on top of it is what took the machine down; it costs nothing to wait the ten minutes.\n\n`--no-reset` keeps the diffs so the judges have something to read.\n\n### 6. Judge — after capture finishes, in small batches\n\nA rubric judge per cell and a comparative reviewer per size, as `/bench-capture` steps 3 and 4 describe. Run them **three or four at a time**, not twelve. Then re-run `capture` once to fold the rubric into the composite.\n\nIf the machine is under load, or the user has other work to do, stop after step 5 and say so: the oracle, wall-clock and test columns are complete without the judges, and the composite can be filled in later.\n\n### 7. Report\n\n```bash\nnode ../speckit-bench/run-all.mjs compare --metric work\nnode ../speckit-bench/run-all.mjs compare --metric oracle\nnode ../speckit-bench/run-all.mjs compare --metric wall\n```\n\nLead with what moved and whether it moved more than the spread of the sweeps before it. A change smaller than that spread has not been measured — say so plainly rather than reporting it as a result.\n\n**Read `work`, not `overall`, when comparing the stock arm to a Companion one.** `overall` gives a quarter of its weight to lifecycle capture, which stock does not have and never will, so it caps stock at 75 by construction. `work` is correctness plus rubric and every arm can reach 100.\n\nThen commit the round in `speckit-bench` — `stats.jsonl`, `history.jsonl`, `runs/`, `reviews/`, `REPORT.md` — with a message naming the sweep and what it was testing.\n\n### If something is incomplete\n\nSay which cells are missing and which columns they cost. A round with eleven of twelve cells and no rubrics is still worth pushing; a round reported as complete when it is not is worse than no round.", "variables": [ "arguments" ], "notes": "Run one full benchmark round on Conduit and report it against previous sweeps Tools: Bash(node ../speckit-bench/sync-templates.mjs:*), Bash(node ../speckit-bench/run-all.mjs:*), Bash(npm run compile:*), Bash(git -C:*), Bash(date:*), Bash(ps:*), Agent, Workflow, AskUserQuestion.", "metadata": { "source": { "provider": "github-claude-commands", "repository": "https://github.com/alfredoperez/speckit-companion", "path": ".claude/commands/bench-experiment.md", "ref": "9fd7ebae23b5badaf5f524105b597ee3c191819e", "url": "https://github.com/alfredoperez/speckit-companion/blob/9fd7ebae23b5badaf5f524105b597ee3c191819e/.claude/commands/bench-experiment.md", "key": "alfredoperez/speckit-companion/.claude/commands/bench-experiment.md" } } }

bench-experiment

About

Your task

1. Name the sweep

2. Check the machine first

3. Bake

4. Prep and drive

5. Measure — with nothing else running

6. Judge — after capture finishes, in small batches

7. Report

If something is incomplete

Install bench-experiment and it becomes one more candidate the router can pick - when it fits.

What it costs to adopt - in engineering time.

One base URL

No vendor lock-in

Failover built in

Use it

Manifest

Reviews