Reference deployments

OpenSmartRoute runs in three shapes. All three use the same Router; only where it lives differs.

Shape	When	How
Library	one Python service, in-process decisions (~0.3 ms / route)	pip install opensmartroute and call Router.route(); see docs/SDK.md
Sidecar	polyglot services, per-pod learner state	run the image next to your app: docker run -p 8000:8000 -v ./examples:/config ghcr.io/isathish/opensmartroute:1.0.0 and call POST /route
Control plane	many callers, shared catalogue, shared learning	the Helm chart in helm/opensmartroute behind a Service/Ingress; state in Redis/SQL via RouterBuilder.with_state_store

Image#

Dockerfile builds a non-root, read-only-rootfs image with the yaml and server extras:

docker build -t opensmartroute:1.0.0 -f deploy/Dockerfile .
docker run --rm -p 8000:8000 -v "$PWD/examples:/config:ro" opensmartroute:1.0.0
curl -s localhost:8000/route -H 'content-type: application/json' -d '{"text":"summarise this contract"}'

Configuration is by environment (OSR_TARGETS, OSR_RULES, OSR_MODELS, OSR_SLM, OSR_STATE, OSR_HOST, OSR_PORT) so the same image serves both sidecar and control-plane roles; entrypoint.sh translates them into osr serve arguments. OSR_SLM points at a routing SLM file from osr slm train; it joins the strategy ensemble and is reported as the slm strategy in every decision trace.

The OpenAI-compatible proxy (/v1/chat/completions) executes LLM targets when the container knows an upstream: set OSR_LLM_BASE_URL (OpenAI, Azure OpenAI, vLLM, Ollama, OpenRouter, LiteLLM ...) and OSR_LLM_API_KEY (or OSR_LLM_API_KEY_ENV naming another variable / <NAME>_FILE mount). Each LLM target's metadata.model is the upstream model name; OSR_LLM_MODEL is the default for targets without one. Without a base URL the proxy still routes but answers 502 for LLM targets.

Access control: the image starts open (every endpoint reachable) so it can act as a sidecar on a private network. To require a token set OSR_SERVER_AUTH_TOKENS=<t1>,<t2> (or mount the list and point OSR_SERVER_AUTH_TOKENS_FILE at it) and, for a hard guarantee, OSR_SERVER_REQUIRE_AUTH=1 so the container refuses to start without one. Mint tokens with osr token generate; clients send Authorization: Bearer <t> or X-API-Key: <t>, and osr login --url https://<host> --token <t> stores one for the CLI (osr whoami then shows edition self-hosted). /healthz, /readyz, /metrics, /whoami and the OpenAPI documents stay open for probes and discovery.

Self-operation: OSR_AUTOPILOT=1 (with OSR_SLM and a writable OSR_STATE) runs the SLM's improvement loop inside the container - every OSR_SLM_AUTOPILOT_INTERVAL_S seconds (default one hour) and early when the drift monitor sees the served success rate drop, it gathers evidence (feedback joined to prompts, cached datasets, seeds), trains a challenger, and hot-swaps it into the ensemble when it beats the champion on the holdout. The promoted model is written to $OSR_STATE/autopilot/slm.json (or OSR_CACHE_DIR), which the entrypoint prefers over OSR_SLM on the next start. OSR_AUTOPILOT_ARGS passes extra flags (--offline, --source NAME, --tier TIER=TARGET, --search QUERY); OSR_CATALOGUE names a catalogue JSON to refresh. GET /stats shows the autopilot status and POST /autopilot/cycle schedules a cycle now.

Observability: every request is traced (spans and events per stage, see docs/OBSERVABILITY.md). GET /healthz is the liveness probe and GET /readyz the readiness probe (503 until the router can score). GET /metrics serves the Prometheus exposition (osr_route_decisions_total{target}, osr_route_latency_ms_bucket, osr_route_policy_rejections_total, osr_route_errors_total, osr_outcome_*, osr_decision_cache_*, a targets gauge, plus per-stage osr_span_duration_ms), GET /trace/{request_id} the full trace of one request (each response carries X-OSR-Trace-Id; send traceparent to join your own trace) and GET /events?name=route.* the recent buffer. Tune with OSR_OBSERVABILITY_MEMORY_EVENTS (buffer size, 0 off), OSR_OBSERVABILITY_LOG_EVENTS=true (JSON lines on stdout), OSR_OBSERVABILITY_EVENTS_FILE (JSONL under the state volume), OSR_OBSERVABILITY_OTEL=true (OpenTelemetry, needs the otel extra and the usual OTEL_EXPORTER_* variables) and OSR_OBSERVABILITY_SAMPLE_RATE under heavy load. The hosted platform image (platform/README.md) reads the same variables and adds the workspace-scoped GET /api/v1/trace/{request_id}, GET /api/v1/events and the public GET /api/v1/status.

Helm#

helm install router deploy/helm/opensmartroute \
  --set-file config.targets=examples/targets.yaml \
  --set-file config.rules=examples/rules.yaml

Key values (see values.yaml for all):

• config.targets / config.rules / config.models -> rendered into a ConfigMap at /config; the pod restarts on change (config checksum annotation).
• autopilot.enabled / autopilot.intervalSeconds / autopilot.args -> self-operation as above (needs config.slm and persistence.enabled). The autopilot promotes models per pod: with replicaCount above 1 each replica trains and serves its own champion, so run one replica (or one autopilot pod behind a shared StateStore) when the loop is on.
• auth.tokens / auth.existingSecret / auth.required -> access tokens for the HTTP API (a chart-managed Secret, or your own with a tokens key), and whether the pod may start without one.
• persistence.enabled -> keep bandit posteriors and the feedback log on a PVC (single replica or RWX class). For several replicas leave it off and use the Redis/SQL StateStore.
• autoscaling, podDisruptionBudget, networkPolicy, ingress -> standard knobs, all off or conservative by default.
• Security: runAsNonRoot, readOnlyRootFilesystem, all capabilities dropped, automountServiceAccountToken: false.

Secrets (MCP manifest signing key OSR_MCP_KEY, encrypted-state key) are injected with env/envFrom secretKeyRefs - never in values.yaml.

Endpoints: GET /healthz, GET /whoami, GET /targets, GET /stats, POST /route, POST /feedback, GET /v1/models, POST /v1/chat/completions (OpenAI-compatible proxy, model: "auto").