--- name: osr-deploy-serve description: Run OpenSmartRoute as a service - the `osr serve` FastAPI app and its /route, /feedback, /targets, /stats, /healthz, /whoami and OpenAI-compatible /v1 endpoints, access tokens for the self-hosted server (`serve --token` / `--generate-token` / `--require-auth`, OSR_SERVER_AUTH_TOKENS, `osr token generate`, `osr login --url ... --token`), the self-operating autopilot (`serve --autopilot`, /autopilot/cycle), the Docker image and entrypoint environment (OSR_TARGETS, OSR_RULES, OSR_STATE, OSR_MODELS, OSR_SLM, OSR_AUTOPILOT, OSR_LLM_BASE_URL, OSR_HOST, OSR_PORT), the Helm chart (auth.tokens), OSR_* settings overrides, state volumes and health checks. Use when deploying, containerising, exposing the router over HTTP, securing it with tokens, or debugging a running instance. license: Apache-2.0 compatibility: OpenSmartRoute >= 0.4, Python >= 3.10, Docker, Kubernetes/Helm 3 metadata: author: opensmartroute osr-domains: "coding general" osr-tags: "opensmartroute deploy docker helm fastapi" osr-quality-prior: "0.85" osr-primary: "false" --- # OpenSmartRoute serve and deploy ## HTTP API (`curl -LsSf https://opensmartroute.ai/install.sh | sh`, or `pip install opensmartroute[server,yaml]`) ``` osr -t targets.yaml [-r rules.yaml] [-s /state] [-m models.json] serve --host 0.0.0.0 --port 8000 osr -t targets.yaml serve --generate-token # prints an osr_local_ token once; all routes need it osr -t targets.yaml serve --require-auth --token $T # production: refuse to start without a token ``` | Method | Path | Body -> response | |---|---|---| | GET | `/healthz` | `{"status": "ok", targets, uptime_s}` (liveness) | | GET | `/readyz` | readiness; 503 until the catalogue has targets and a probe request routes end to end | | GET | `/metrics` | Prometheus text: `MetricsTelemetry.prometheus()` (`osr_route_decisions_total{target}`, `osr_route_latency_ms_bucket`, `osr_outcome_*`, `osr_decision_cache_*`) + tracer counters + `osr_targets` | | GET | `/whoami` | `{service, edition: "self-hosted", version, auth_required, authenticated, targets}` - what `osr whoami` prints | | GET | `/targets` | catalogue (`registry.to_dicts()`) | | GET | `/stats` | feedback aggregates (+ `health` from an EnterpriseRouter, + `autopilot` status when enabled) | | POST | `/route` | `{text, context, history, objective, constraints, kinds, top_k=3, plan=false}` -> `RouteDecision.to_dict()`; 422 when no route | | POST | `/feedback` | `{request_id, target_id, success, quality, cost_usd, latency_ms, domains, complexity, preferred_over}` -> `{"status":"recorded"}` | | POST | `/autopilot/cycle` | (with `--autopilot`) schedule an improvement cycle now -> `{scheduled, status}` | | GET | `/v1/models` | OpenAI list: `auto` + every primary target id | | POST | `/v1/chat/completions` | OpenAI body; `model: "auto"` or `"opensmartroute"` routes (plan on), a target id pins; 400 stream, 404 unknown model, 422 no route, 502 execution error | Python: `from opensmartroute.server import create_app; app = create_app(router, proxy_aliases=..., autopilot=..., auth_tokens=[...])` - mount it inside your own FastAPI app or pass an `EnterpriseRouter`-built `router`. ## Access tokens - Open by default (sidecar on a private network). With one or more tokens (`--token`, `--generate-token`, `OSR_SERVER_AUTH_TOKENS=a,b`, `OSR_SERVER_AUTH_TOKENS_FILE=/run/secrets/tokens`, `ServerSettings.auth_tokens`) every path except `/healthz`, `/readyz`, `/metrics`, `/whoami`, `/docs`, `/openapi.json`, `/redoc` needs `Authorization: Bearer ` or `X-API-Key: ` (`server.OPEN_PATHS`, constant-time compare); otherwise 401 + `WWW-Authenticate: Bearer`. `--require-auth` / `OSR_SERVER_REQUIRE_AUTH=1` fails start-up without a token. - Tokens are opaque strings; `osr token generate [--count N]` mints `osr_local_...` ones (`credentials.generate_token`). Sign a CLI in with `osr login --url http://host:8000 --token ` (stored per profile under `~/.config/opensmartroute/credentials.json`, `OSR_CONFIG_DIR` overrides); `osr whoami` calls `/whoami`. There is no account or key registry on a self-hosted server - rotation is "add the new token, restart, remove the old one". - Helm: `auth.tokens: [..]` (chart-managed Secret), `auth.existingSecret` (key `tokens`, comma separated), `auth.required`. Docker: pass the same `OSR_SERVER_*` variables; the entrypoint needs no change. ## Self-operation (`serve --autopilot`) ``` osr -t targets.yaml --slm slm.json -s /state serve --autopilot [--autopilot-interval 3600] \ [--cache-dir DIR] [--catalogue cat.json] [--source NAME ...] [--tier TIER=TARGET ...] [--search Q ...] [--offline] ``` `learning.Autopilot` runs `SelfImprover` cycles in a daemon thread: on the schedule (`OSR_SLM_AUTOPILOT_INTERVAL_S`), early when `DriftMonitor` (Page-Hinkley on outcome success / quality, fed through `Router.observers`) alarms, or on `POST /autopilot/cycle`; never closer than `OSR_SLM_AUTOPILOT_MIN_GAP_S`. Feedback becomes training rows through the router's request memory (`OSR_SLM_AUTOPILOT_REMEMBER`); an accepted challenger is hot-swapped into the served `SLMStrategy` and saved (to the served file when writable, else `/slm.json`). A failing cycle is counted in `/stats` -> `autopilot.errors`, never fatal. Container: `OSR_AUTOPILOT=1` (+ `OSR_CACHE_DIR`, `OSR_CATALOGUE`, `OSR_AUTOPILOT_ARGS="--offline ..."`); the entrypoint prefers a previously promoted `$OSR_STATE/autopilot/slm.json` over `OSR_SLM` on restart. Helm: `autopilot.enabled`, `autopilot.intervalSeconds`, `autopilot.args` (needs `config.slm` + `persistence.enabled`). ## Container ```dockerfile # deploy/Dockerfile installs .[yaml,server,fast] (numpy: autopilot retrains 10x faster); defaults OSR_TARGETS=/config/targets.yaml OSR_STATE=/state; EXPOSE 8000 docker build -t opensmartroute -f deploy/Dockerfile . docker run -p 8000:8000 -v $PWD/examples:/config:ro -v osr-state:/state \ -e OSR_TARGETS=/config/targets.yaml -e OSR_RULES=/config/rules.yaml opensmartroute ``` `deploy/entrypoint.sh` reads `OSR_TARGETS` (required), `OSR_RULES`, `OSR_STATE`, `OSR_MODELS`, `OSR_SLM` (routing SLM from `osr slm train`, joins the ensemble as the `slm` strategy), `OSR_HOST` (default `0.0.0.0`), `OSR_PORT` (default `8000`) and execs `osr ... serve`. Health check hits `/healthz`. Mount `/state` on a persistent volume or learner state is lost on restart. The `/v1/chat/completions` proxy only *executes* when LLM targets have handlers. Set `OSR_LLM_BASE_URL` to any OpenAI-compatible endpoint (OpenAI, Azure OpenAI, vLLM, Ollama at `http://host.docker.internal:11434/v1`, OpenRouter, LiteLLM) plus `OSR_LLM_API_KEY` (or `OSR_LLM_API_KEY_ENV=`); each LLM target's `metadata.model` is the upstream model name and `OSR_LLM_MODEL` the default for targets without one. Without it the proxy still routes but LLM targets answer 502 "has no handler". Verified end to end: `X-OSR-Target` / `X-OSR-Request-Id` headers, `model: "auto"`, `"auto:cheap"`, a pinned target id, 404 unknown model, 400 stream. ## Helm ``` helm install osr deploy/helm/opensmartroute -f values.yaml ``` `values.yaml` covers image, replicas, HPA, ingress, service account, a ConfigMap for `targets.yaml`/`rules.yaml`, extra env/`envFrom` for secrets (`OSR_MCP_KEY`, encrypted-state key) and a PVC for `OSR_STATE`. With more than one replica use a shared `StateStore` (Redis/SQL) via a small Python entrypoint, or run one writer and read-only replicas. ## Tuning without code Every tunable is an `OSR__` variable - run `osr settings` to list them with current values. Common ones: `OSR_ROUTING_SOFTMAX_TEMPERATURE`, `OSR_ROUTING_NARROW_ABOVE`, `OSR_POLICY_JAILBREAK_THRESHOLD`, `OSR_WEIGHTS_RULES`, `OSR_WEIGHTS_CAPABILITY`, `OSR_BANDIT_CONFIDENCE_WINDOW`. Bad values fail fast at startup with `ConfigurationError`. ## Operating - Route latency is sub-millisecond for local strategies; only an LLM judge adds network time - keep `escalate_llm_judge_below` low and use `TimeoutMiddleware`. - Post `Outcome`s to `/feedback` from the caller that observed the result; `/stats` shows them. - Watch `route_latency_ms.p95`, `policy_rejections_total`, `errors_by_type` from `MetricsTelemetry.snapshot()`; `GET /metrics` renders the same store as Prometheus text (`MetricsTelemetry.prometheus(namespace="osr", extra=...)`). Scrape it with a `ServiceMonitor` or `prometheus.io/scrape` annotations. `RouterBuilder.with_cache(ttl_s, max_size)` adds a decision cache whose hits and misses appear as `osr_decision_cache_*`; `with_audit(sink, outcomes=True)` chains outcomes too. - Tracing: `GET /metrics` also carries the tracer's `osr_spans_total` / `osr_span_duration_ms` per stage, `GET /trace/{request_id}` returns every span and event of one request (`X-OSR-Trace-Id` on each response; send `traceparent` to join your own trace), `GET /events?name=route.*` filters the buffer. `OSR_OBSERVABILITY_LOG_EVENTS=true` for JSON lines, `OSR_OBSERVABILITY_EVENTS_FILE=/state/events.jsonl` for a file, `OSR_OBSERVABILITY_OTEL=true` (+ `otel` extra) for OpenTelemetry, `OSR_OBSERVABILITY_SAMPLE_RATE` under load. See `docs/OBSERVABILITY.md`. The hosted platform image reads the same variables; there the buffer is workspace-scoped (`GET /api/v1/trace/{request_id}` with outcomes and audit records, `GET /api/v1/events`, public `GET /api/v1/status` for status pages) - `osr-platform` skill. - Use `osr -t targets.yaml eval data.jsonl --min-accuracy 0.9` in the image build pipeline so a catalogue change cannot ship a regression. - Secrets: env or `_FILE` mounts via `load_secret`; never bake them into the image or YAML.