osr-deploy-serve

HTTP API (curl -LsSf https://opensmartroute.ai/install.sh | sh, or pip install opensmartroute[server,yaml])#

osr -t targets.yaml [-r rules.yaml] [-s /state] [-m models.json] serve --host 0.0.0.0 --port 8000
osr -t targets.yaml serve --generate-token            # prints an osr_local_ token once; all routes need it
osr -t targets.yaml serve --require-auth --token $T   # production: refuse to start without a token

Method	Path	Body -> response
GET	/healthz	{"status": "ok", targets, uptime_s} (liveness)
GET	/readyz	readiness; 503 until the catalogue has targets and a probe request routes end to end
GET	/metrics	Prometheus text: MetricsTelemetry.prometheus() (osr_route_decisions_total{target}, osr_route_latency_ms_bucket, osr_outcome_*, osr_decision_cache_*) + tracer counters + osr_targets
GET	/whoami	{service, edition: "self-hosted", version, auth_required, authenticated, targets} - what osr whoami prints
GET	/targets	catalogue (registry.to_dicts())
GET	/stats	feedback aggregates (+ health from an EnterpriseRouter, + autopilot status when enabled)
POST	/route	{text, context, history, objective, constraints, kinds, top_k=3, plan=false} -> RouteDecision.to_dict(); 422 when no route
POST	/feedback	{request_id, target_id, success, quality, cost_usd, latency_ms, domains, complexity, preferred_over} -> {"status":"recorded"}
POST	/autopilot/cycle	(with --autopilot) schedule an improvement cycle now -> {scheduled, status}
GET	/v1/models	OpenAI list: auto + every primary target id
POST	/v1/chat/completions	OpenAI body; model: "auto" or "opensmartroute" routes (plan on), a target id pins; 400 stream, 404 unknown model, 422 no route, 502 execution error

Python: from opensmartroute.server import create_app; app = create_app(router, proxy_aliases=..., autopilot=..., auth_tokens=[...])

• mount it inside your own FastAPI app or pass an EnterpriseRouter-built router.

Access tokens#

• Open by default (sidecar on a private network). With one or more tokens (--token, --generate-token, OSR_SERVER_AUTH_TOKENS=a,b, OSR_SERVER_AUTH_TOKENS_FILE=/run/secrets/tokens, ServerSettings.auth_tokens) every path except /healthz, /readyz, /metrics, /whoami, /docs, /openapi.json, /redoc needs Authorization: Bearer <t> or X-API-Key: <t> (server.OPEN_PATHS, constant-time compare); otherwise 401 + WWW-Authenticate: Bearer. --require-auth / OSR_SERVER_REQUIRE_AUTH=1 fails start-up without a token.
• Tokens are opaque strings; osr token generate [--count N] mints osr_local_... ones (credentials.generate_token). Sign a CLI in with osr login --url http://host:8000 --token <t> (stored per profile under ~/.config/opensmartroute/credentials.json, OSR_CONFIG_DIR overrides); osr whoami calls /whoami. There is no account or key registry on a self-hosted server - rotation is "add the new token, restart, remove the old one".
• Helm: auth.tokens: [..] (chart-managed Secret), auth.existingSecret (key tokens, comma separated), auth.required. Docker: pass the same OSR_SERVER_* variables; the entrypoint needs no change.

Self-operation (serve --autopilot)#

osr -t targets.yaml --slm slm.json -s /state serve --autopilot [--autopilot-interval 3600] \
    [--cache-dir DIR] [--catalogue cat.json] [--source NAME ...] [--tier TIER=TARGET ...] [--search Q ...] [--offline]

learning.Autopilot runs SelfImprover cycles in a daemon thread: on the schedule (OSR_SLM_AUTOPILOT_INTERVAL_S), early when DriftMonitor (Page-Hinkley on outcome success / quality, fed through Router.observers) alarms, or on POST /autopilot/cycle; never closer than OSR_SLM_AUTOPILOT_MIN_GAP_S. Feedback becomes training rows through the router's request memory (OSR_SLM_AUTOPILOT_REMEMBER); an accepted challenger is hot-swapped into the served SLMStrategy and saved (to the served file when writable, else <cache-dir>/slm.json). A failing cycle is counted in /stats -> autopilot.errors, never fatal. Container: OSR_AUTOPILOT=1 (+ OSR_CACHE_DIR, OSR_CATALOGUE, OSR_AUTOPILOT_ARGS="--offline ..."); the entrypoint prefers a previously promoted $OSR_STATE/autopilot/slm.json over OSR_SLM on restart. Helm: autopilot.enabled, autopilot.intervalSeconds, autopilot.args (needs config.slm + persistence.enabled).

Container#

# deploy/Dockerfile installs .[yaml,server,fast] (numpy: autopilot retrains 10x faster); defaults OSR_TARGETS=/config/targets.yaml OSR_STATE=/state; EXPOSE 8000
docker build -t opensmartroute -f deploy/Dockerfile .
docker run -p 8000:8000 -v $PWD/examples:/config:ro -v osr-state:/state \
  -e OSR_TARGETS=/config/targets.yaml -e OSR_RULES=/config/rules.yaml opensmartroute

deploy/entrypoint.sh reads OSR_TARGETS (required), OSR_RULES, OSR_STATE, OSR_MODELS, OSR_SLM (routing SLM from osr slm train, joins the ensemble as the slm strategy), OSR_HOST (default 0.0.0.0), OSR_PORT (default 8000) and execs osr ... serve. Health check hits /healthz. Mount /state on a persistent volume or learner state is lost on restart.

The /v1/chat/completions proxy only executes when LLM targets have handlers. Set OSR_LLM_BASE_URL to any OpenAI-compatible endpoint (OpenAI, Azure OpenAI, vLLM, Ollama at http://host.docker.internal:11434/v1, OpenRouter, LiteLLM) plus OSR_LLM_API_KEY (or OSR_LLM_API_KEY_ENV=<VAR>); each LLM target's metadata.model is the upstream model name and OSR_LLM_MODEL the default for targets without one. Without it the proxy still routes but LLM targets answer 502 "has no handler". Verified end to end: X-OSR-Target / X-OSR-Request-Id headers, model: "auto", "auto:cheap", a pinned target id, 404 unknown model, 400 stream.

Helm#

helm install osr deploy/helm/opensmartroute -f values.yaml

values.yaml covers image, replicas, HPA, ingress, service account, a ConfigMap for targets.yaml/rules.yaml, extra env/envFrom for secrets (OSR_MCP_KEY, encrypted-state key) and a PVC for OSR_STATE. With more than one replica use a shared StateStore (Redis/SQL) via a small Python entrypoint, or run one writer and read-only replicas.

Tuning without code#

Every tunable is an OSR_<GROUP>_<FIELD> variable - run osr settings to list them with current values. Common ones: OSR_ROUTING_SOFTMAX_TEMPERATURE, OSR_ROUTING_NARROW_ABOVE, OSR_POLICY_JAILBREAK_THRESHOLD, OSR_WEIGHTS_RULES, OSR_WEIGHTS_CAPABILITY, OSR_BANDIT_CONFIDENCE_WINDOW. Bad values fail fast at startup with ConfigurationError.

Operating#

• Route latency is sub-millisecond for local strategies; only an LLM judge adds network time - keep escalate_llm_judge_below low and use TimeoutMiddleware.
• Post Outcomes to /feedback from the caller that observed the result; /stats shows them.
• Watch route_latency_ms.p95, policy_rejections_total, errors_by_type from MetricsTelemetry.snapshot(); GET /metrics renders the same store as Prometheus text (MetricsTelemetry.prometheus(namespace="osr", extra=...)). Scrape it with a ServiceMonitor or prometheus.io/scrape annotations. RouterBuilder.with_cache(ttl_s, max_size) adds a decision cache whose hits and misses appear as osr_decision_cache_*; with_audit(sink, outcomes=True) chains outcomes too.
• Tracing: GET /metrics also carries the tracer's osr_spans_total / osr_span_duration_ms per stage, GET /trace/{request_id} returns every span and event of one request (X-OSR-Trace-Id on each response; send traceparent to join your own trace), GET /events?name=route.* filters the buffer. OSR_OBSERVABILITY_LOG_EVENTS=true for JSON lines, OSR_OBSERVABILITY_EVENTS_FILE=/state/events.jsonl for a file, OSR_OBSERVABILITY_OTEL=true (+ otel extra) for OpenTelemetry, OSR_OBSERVABILITY_SAMPLE_RATE under load. See docs/OBSERVABILITY.md. The hosted platform image reads the same variables; there the buffer is workspace-scoped (GET /api/v1/trace/{request_id} with outcomes and audit records, GET /api/v1/events, public GET /api/v1/status for status pages) - osr-platform skill.
• Use osr -t targets.yaml eval data.jsonl --min-accuracy 0.9 in the image build pipeline so a catalogue change cannot ship a regression.
• Secrets: env or <NAME>_FILE mounts via load_secret; never bake them into the image or YAML.