Reference deployments

OpenSmartRoute runs in three shapes. All three use the same Router; only where it lives differs.

Image

Dockerfile builds a non-root, read-only-rootfs image with the yaml and server extras:

docker build -t opensmartroute:2.1.0 -f deploy/Dockerfile .
docker run --rm -p 8000:8000 opensmartroute:2.1.0                              # built-in starter catalogue
docker run --rm -p 8000:8000 -v "$PWD/examples:/config:ro" opensmartroute:2.1.0   # your targets.yaml / rules.yaml
curl -s localhost:8000/route -H 'content-type: application/json' -d '{"text":"summarise this contract"}'

With nothing mounted at /config/targets.yaml the container serves the CLI's built-in starter catalogue (llm-small, llm-frontier, llm-onprem) and says so in its log; mount a folder with targets.yaml (and rules.yaml, set OSR_RULES=/config/rules.yaml) to replace it.

Configuration is by environment (OSR_TARGETS, OSR_RULES, OSR_MODELS, OSR_SLM, OSR_STATE, OSR_HOST, OSR_PORT) so the same image serves both sidecar and control-plane roles; entrypoint.sh translates them into osr serve arguments. OSR_SLM points at a routing SLM file from osr slm train (or bundled for the model shipped inside the package); it joins the strategy ensemble and is reported as the slm strategy in every decision trace.

The OpenAI-compatible proxy (/v1/chat/completions) executes LLM targets when the container knows an upstream: set OSR_LLM_BASE_URL (OpenAI, Azure OpenAI, vLLM, Ollama, OpenRouter, LiteLLM ...) and OSR_LLM_API_KEY (or OSR_LLM_API_KEY_ENV naming another variable / <NAME>_FILE mount). Each LLM target's metadata.model is the upstream model name; OSR_LLM_MODEL is the default for targets without one. Targets tagged embeddings get an embeddings handler instead and answer /v1/embeddings; targets tagged transcription / speech and image tools (actions: [image]) with a metadata.model get the media handlers and answer /v1/audio/transcriptions, /v1/audio/speech and /v1/images/generations on the same provider; /v1/responses (the OpenAI Responses API) and /v1/messages (the Anthropic Messages API, for Claude Code and the Anthropic SDKs) ride on the chat path. Without a base URL the proxy still routes but answers 502 for LLM targets.

Access control: the image starts open (every endpoint reachable) so it can act as a sidecar on a private network. To require a token set OSR_SERVER_AUTH_TOKENS=<t1>,<t2> (or mount the list and point OSR_SERVER_AUTH_TOKENS_FILE at it) and, for a hard guarantee, OSR_SERVER_REQUIRE_AUTH=1 so the container refuses to start without one. Mint tokens with osr token generate; clients send Authorization: Bearer <t> or X-API-Key: <t>, and osr login --url https://<host> --token <t> stores one for the CLI (osr whoami then shows edition self-hosted). A <t>:ro entry is a read-only token (dashboards, CI): it can read everything and ask for decisions but cannot teach the learners or run a target. /healthz, /readyz, /metrics, /whoami and the OpenAPI documents stay open for probes and discovery.

Hardening: the served router takes the production controls from the environment (each one is also a serve --... flag) - OSR_SERVER_GUARD=1 (input guard: gadget / steering detection, size limits), OSR_SERVER_REDACT_PII=1 (PII redacted before routing, originals only reach the executor), OSR_SERVER_OUTPUT_GUARD=flag|redact|block (every answer screened for leaked secrets, PII and instruction injection), OSR_SERVER_HEALTH=1 (circuit breakers per target), OSR_SERVER_AUDIT_PATH=/state/audit.jsonl (hash-chained decision trail), OSR_SERVER_ORIGIN_POLICY=1 (tool-parameter provenance on state-changing tools), OSR_SERVER_RESOURCE_LIMITS=1 (per-task step / call / token / cost / wall-clock ceilings) and OSR_SERVER_STATE_STORE=redis://...|postgresql://...|sqlite:///state/learners.db (+ OSR_SERVER_STATE_KEY_ENV=<VAR> to encrypt it) for the learner state instead of files under OSR_STATE. Set them in the chart's env; the container reports Hardening: ... on start.

Self-operation: OSR_AUTOPILOT=1 (with OSR_SLM and a writable OSR_STATE) runs the SLM's improvement loop inside the container - every OSR_SLM_AUTOPILOT_INTERVAL_S seconds (default one hour) and early when the drift monitor sees the served success rate drop, it gathers evidence (feedback joined to prompts, cached datasets, seeds), trains a challenger, and hot-swaps it into the ensemble when it beats the champion on the holdout. The promoted model is written to $OSR_STATE/autopilot/slm.json (or OSR_CACHE_DIR), which the entrypoint prefers over OSR_SLM on the next start. OSR_AUTOPILOT_ARGS passes extra flags (--offline, --source NAME, --tier TIER=TARGET, --search QUERY, --feed URL); OSR_CATALOGUE names a catalogue JSON to refresh; OSR_SLM_FEEDS=https://...,https://... lists article feeds (vendor blogs, release notes) the loop reads for new models. GET /stats shows the autopilot status and POST /autopilot/cycle schedules a cycle now.

Following the marketplace: OSR_MARKETPLACE_REFRESH_S=86400 (with the optional OSR_MARKETPLACE_KINDS, OSR_MARKETPLACE_QUERY, OSR_MARKETPLACE_TAG, OSR_MARKETPLACE_LIMIT, OSR_MARKETPLACE_SLUGS, OSR_MARKETPLACE_URL) makes the server upsert the followed listings into its catalogue on that schedule - a listing the marketplace dropped is removed, a target you edited locally is kept, and OSR_MARKETPLACE_PRUNE=1 also removes what no sync lists any more; GET /stats shows the marketplace status and POST /marketplace/sync runs one now. Prompt compaction before every model call is on by default (OSR_COMPRESSION_ENABLED=0 turns it off; OSR_COMPRESSION_HISTORY_BUDGET_TOKENS adds a conversation budget); tool slots in plans are governed by OSR_ROUTING_TOOL_SLOTS (0 = never offer tools).

Observability: every request is traced (spans and events per stage, see Tracing and observability). GET /healthz is the liveness probe (the process is up) and GET /readyz the readiness probe (503 until the catalogue is loaded and a probe request routes end to end). The image's Docker HEALTHCHECK and the Helm chart's readinessProbe use /readyz, so a container is only reported healthy / receives traffic once it can route; the chart's livenessProbe stays on /healthz so a slow catalogue load never gets the pod killed. GET /metrics serves the Prometheus exposition (osr_route_decisions_total{target}, osr_route_latency_ms_bucket, osr_route_policy_rejections_total, osr_route_errors_total, osr_outcome_*, osr_decision_cache_*, a targets gauge, plus per-stage osr_span_duration_ms), GET /trace/{request_id} the full trace of one request (each response carries X-OSR-Trace-Id; send traceparent to join your own trace) and GET /events?name=route.* the recent buffer. Tune with OSR_OBSERVABILITY_MEMORY_EVENTS (buffer size, 0 off), OSR_OBSERVABILITY_LOG_EVENTS=true (JSON lines on stdout), OSR_OBSERVABILITY_EVENTS_FILE (JSONL under the state volume), OSR_OBSERVABILITY_OTEL=true (OpenTelemetry, needs the otel extra and the usual OTEL_EXPORTER_* variables) and OSR_OBSERVABILITY_SAMPLE_RATE under heavy load. The hosted platform image (platform/README.md) reads the same variables and adds the workspace-scoped GET /api/v1/trace/{request_id}, GET /api/v1/events and the public GET /api/v1/status.

Helm

The chart is published with every release (https://opensmartroute.ai/download/helm, the helm package tarball listed with its SHA-256 on the downloads page):

curl -fsSLO https://opensmartroute.ai/download/helm            # opensmartroute-<version>.tgz
helm install router opensmartroute-*.tgz \
  --set-file config.targets=targets.yaml \
  --set-file config.rules=rules.yaml

Key values (see values.yaml for all):

The chart refuses two installs that look fine and are not, naming what to set:

• no catalogue. config.targets is empty by default and there is no illustrative one behind it, because a router installed with invented targets and invented prices would price and choose real traffic against them. Pass your own (--set-file config.targets=targets.yaml) or point config.existingConfigMap at one.

• an unauthenticated API on a hostname. ingress.enabled without auth.tokens or auth.existingSecret publishes an open router. Mint a token with osr token generate, or set auth.allowAnonymousIngress=true if that is genuinely what you want.

• config.targets / config.rules / config.models -> rendered into a ConfigMap at /config; the pod restarts on change (config checksum annotation).

• autopilot.enabled / autopilot.intervalSeconds / autopilot.args -> self-operation as above (needs persistence.enabled; config.slm is an optional warm start, otherwise the loop trains a fresh model into /state/autopilot/slm.json on its first cycle). The autopilot promotes models per pod: with replicaCount above 1 each replica trains and serves its own champion, so run one replica (or one autopilot pod behind a shared StateStore) when the loop is on. autopilot.feeds -> article feeds the loop reads (OSR_SLM_FEEDS).

• marketplace.refreshSeconds (+ url, kinds, query, tag, limit, slugs) -> follow the marketplace (OSR_MARKETPLACE_*): new and re-versioned skills and tools land in the served catalogue on that schedule.

• auth.tokens / auth.existingSecret / auth.required / auth.allowAnonymousIngress -> access tokens for the HTTP API (a chart-managed Secret, or your own with a tokens key), whether the pod may start without one, and the opt-out that lets an ingress publish an open router on purpose.

• hardening.guard / learnedGuard / redactPii / outputGuard (off|flag|redact|block) / health / auditPath / originPolicy / resourceLimits -> the production controls above (OSR_SERVER_*), all off by default; put auditPath under /state with persistence.enabled so the chain survives restarts.

• persistence.enabled -> keep bandit posteriors and the feedback log on a PVC (single replica or RWX class). For several replicas leave it off and use the Redis/SQL StateStore.

• autoscaling, podDisruptionBudget, networkPolicy, ingress -> standard knobs, all off or conservative by default.

• livenessProbe (/healthz) / readinessProbe (/readyz) / startupProbe (/healthz, five minutes of grace so a large catalogue, slow persistent storage or an autopilot cold start is not killed mid-initialisation) -> probe paths and timings; CI lints the chart, renders both the default and the fully-enabled value sets through kubeconform, and installs the chart with the freshly built image on a kind cluster (helm install --wait, /readyz through the Service, the chart's token gating /targets and /route).

• Security: runAsNonRoot, readOnlyRootFilesystem, all capabilities dropped, automountServiceAccountToken: false.

Secrets (MCP manifest signing key OSR_MCP_KEY, encrypted-state key) are injected with env/envFrom secretKeyRefs - never in values.yaml.

Endpoints: GET /healthz, GET /readyz, GET /metrics, GET /whoami, GET /targets, GET /stats, POST /route, POST /feedback, GET /v1/models, POST /v1/chat/completions (OpenAI-compatible proxy, model: "osr/auto", streaming and tool calling), POST /v1/embeddings, POST /mcp (Model Context Protocol).