osr-deploy-serve

HTTP API (curl -LsSf https://opensmartroute.ai/install.sh | sh, or pip install opensmartroute[server,yaml])

osr -t targets.yaml [-r rules.yaml] [-s /state] [-m models.json] serve --host 0.0.0.0 --port 8000
osr -t targets.yaml serve --generate-token            # prints an osr_local_ token once; all routes need it
osr -t targets.yaml serve --require-auth --token $T   # production: refuse to start without a token
osr -t targets.yaml serve --guard --redact-pii --output-guard redact   # screen prompts, mask PII, rewrite leaked secrets / PII in answers

Guards on the served router: --guard (input guard: gadget / steering detection, size limits; --learned-guard swaps in the trained detector), --redact-pii (implies --guard), --output-guard off|flag|redact|block (the answer side: leaked secrets, PII, relayed injection; OSR_SERVER_OUTPUT_GUARD, default off on osr serve). Verdicts become guard.* / output_guard.* trace events and output_guard metadata on proxied responses. --web-grounding (OSR_SERVER_WEB_GROUNDING=1) searches the web for requests about current facts (signals.freshness_need) and hands the injection-screened hits to the model as a Live web context system block (web.grounded / web.empty / web.failed events).

Code next to the catalogue: osr --components my_components serve (repeatable; module:setup also calls setup(registry)) or OSR_SERVER_COMPONENTS=pkg.module,pkg.other:setup imports Python modules whose @tool / @handler / rule() / @guard / @on_outcome declarations (osr-decorator-sdk) join the served router - handlers bound to catalogue targets, rules merged, guards and hooks running through the hardened router. The module must be importable from the server's PYTHONPATH (mount it next to /config in the container).

Python: from opensmartroute.server import create_app; app = create_app(router, proxy_aliases=..., autopilot=..., auth_tokens=[...])

• mount it inside your own FastAPI app or pass an EnterpriseRouter-built router.

Access tokens

• Open by default (sidecar on a private network). With one or more tokens (--token, --generate-token, OSR_SERVER_AUTH_TOKENS=a,b, OSR_SERVER_AUTH_TOKENS_FILE=/run/secrets/tokens, ServerSettings.auth_tokens) every path except /healthz, /readyz, /metrics, /whoami, /docs, /openapi.json, /redoc needs Authorization: Bearer <t> or X-API-Key: <t> (server.OPEN_PATHS, constant-time compare); otherwise 401 + WWW-Authenticate: Bearer. --require-auth / OSR_SERVER_REQUIRE_AUTH=1 fails start-up without a token.
• Tokens are opaque strings; osr token generate [--count N] mints osr_local_... ones (credentials.generate_token). Sign a CLI in with osr login --url http://host:8000 --token <t> (stored per profile under ~/.config/opensmartroute/credentials.json, OSR_CONFIG_DIR overrides); osr whoami calls /whoami. There is no account or key registry on a self-hosted server - rotation is "add the new token, restart, remove the old one".
• Helm: auth.tokens: [..] (chart-managed Secret), auth.existingSecret (key tokens, comma separated), auth.required. Docker: pass the same OSR_SERVER_* variables; the entrypoint needs no change.

Self-operation (serve --autopilot)

osr -t targets.yaml [--slm slm.json] -s /state serve --autopilot [--autopilot-interval 3600] \
    [--cache-dir DIR] [--catalogue cat.json] [--source NAME ...] [--tier TIER=TARGET ...] [--search Q ...] \
    [--feed URL ...] [--offline] [--marketplace-refresh 86400 [--marketplace-url URL]]

--slm is optional: without it the loop starts warm - the model shipped inside the package (--slm bundled names it explicitly; RouterSLM.bundled()) over the catalogue, or the model a previous run promoted into <cache-dir>/slm.json, joins the ensemble and the first cycle trains a challenger on your rows. learning.Autopilot runs SelfImprover cycles in a daemon thread: on the schedule (OSR_SLM_AUTOPILOT_INTERVAL_S), early when DriftMonitor (Page-Hinkley on outcome success / quality, fed through Router.observers) alarms, or on POST /autopilot/cycle; never closer than OSR_SLM_AUTOPILOT_MIN_GAP_S. Feedback becomes training rows through the router's request memory (OSR_SLM_AUTOPILOT_REMEMBER); an accepted challenger is hot-swapped into the served SLMStrategy and saved (to the served file when writable, else <cache-dir>/slm.json). A failing cycle is counted in /stats -> autopilot.errors, never fatal. Container: OSR_AUTOPILOT=1 (+ OSR_CACHE_DIR, OSR_CATALOGUE, OSR_AUTOPILOT_ARGS="--offline ..."); the entrypoint prefers a previously promoted $OSR_STATE/autopilot/slm.json over OSR_SLM on restart, and OSR_SLM may stay empty. Helm: autopilot.enabled, autopilot.intervalSeconds, autopilot.args (needs persistence.enabled; config.slm is an optional warm start). --feed URL / OSR_SLM_FEEDS add RSS / Atom article feeds (vendor blogs, release notes) the cycle reads: mentioned catalogue models become trending candidates, unknown announced names become card lookups.

Following the marketplace and compacting prompts

serve --marketplace-refresh SECONDS (or OSR_MARKETPLACE_REFRESH_S) starts adapters.MarketplaceSync: the followed listings (OSR_MARKETPLACE_KINDS default skill,tool, OSR_MARKETPLACE_QUERY, OSR_MARKETPLACE_TAG, OSR_MARKETPLACE_LIMIT, OSR_MARKETPLACE_SLUGS, OSR_MARKETPLACE_URL) are upserted into the served catalogue on that schedule - no restart, no cron. GET /stats -> marketplace, POST /marketplace/sync (write path; read-only tokens get 403) runs one now. Helm: marketplace.refreshSeconds (+ url, kinds, query, tag, limit, slugs) and autopilot.feeds. Every model call is compacted first (OSR_COMPRESSION_*; whitespace, repeated lines, optional history budget; fenced code untouched) and the proxy metadata reports tokens_saved. Plans offer tool targets to the model as functions and run its calls (OSR_ROUTING_TOOL_SLOTS, default 2; 0 turns it off).

Container

# deploy/Dockerfile installs .[yaml,server,fast] (numpy: autopilot retrains 10x faster); defaults OSR_TARGETS=/config/targets.yaml OSR_STATE=/state; EXPOSE 8000
docker build -t opensmartroute -f deploy/Dockerfile .
docker run -p 8000:8000 -v $PWD/examples:/config:ro -v osr-state:/state \
  -e OSR_TARGETS=/config/targets.yaml -e OSR_RULES=/config/rules.yaml opensmartroute

deploy/entrypoint.sh reads OSR_TARGETS (required), OSR_RULES, OSR_STATE, OSR_MODELS, OSR_SLM (routing SLM from osr slm train, joins the ensemble as the slm strategy), OSR_HOST (default 0.0.0.0), OSR_PORT (default 8000) and execs osr ... serve. Health check hits /healthz. Mount /state on a persistent volume or learner state is lost on restart.

The /v1/chat/completions proxy only executes when LLM targets have handlers. Set OSR_LLM_BASE_URL to any OpenAI-compatible endpoint (OpenAI, Azure OpenAI, vLLM, Ollama at http://host.docker.internal:11434/v1, OpenRouter, LiteLLM) plus OSR_LLM_API_KEY (or OSR_LLM_API_KEY_ENV=<VAR>); each LLM target's metadata.model is the upstream model name and OSR_LLM_MODEL the default for targets without one. Without it the proxy still routes but LLM targets answer 502 "has no handler". Verified end to end: X-OSR-Target / X-OSR-Request-Id headers, model: "auto", "auto:cheap", a pinned target id, 404 unknown model, 400 stream.

Sizing a self-hosted model server (CPU / RAM)

opensmartroute.inference (osr capacity) turns "does this model fit and how fast will it answer" into arithmetic before a deploy: weights = parameters x bits per weight (Q4_K_M ~4.85, Q8_0 8.5, F16 16, MXFP4 4.25); KV cache = 2 x layers x KV heads x head size x bytes (f16 2, q8_0 ~1) x OLLAMA_CONTEXT_LENGTH x OLLAMA_NUM_PARALLEL; plus ~10 % runtime overhead, minus 12 % host headroom. Decode is bandwidth bound (tokens/s ~ GB/s / active bytes per token - an MoE reads only its active experts), prefill compute bound (2 FLOPs per active parameter per token), a cold load = weights / store read rate (a network share: 0.05-0.3 GB/s).

osr capacity show qwen3.5:4b glm-4.7-flash --memory 28Gi --cores 4    # footprint, fits?, tok/s, warm vs cold latency
osr capacity plan qwen3.5:4b qwen2.5-coder:7b --memory 8Gi --hosts 2  # first-fit-decreasing packing + OLLAMA_* env
osr capacity plan glm-4.7-flash --memory 14Gi --shrink --strict       # coarser KV / shorter ctx / fewer slots; exit 2 if unplaced
osr capacity show --url http://ollama:11434 --memory 28Gi             # exact shapes from /api/show and /api/tags
osr capacity ps --url http://ollama:11434                             # /api/ps: loaded models, CPU or GPU, resident or expiring

ModelSpec.from_tag reads size / -a3b active share / quant from an Ollama tag (KNOWN_TAGS covers names without a size: gpt-oss, glm-4.7-flash, glm-ocr); OllamaClient(url).specs() gets the exact shape. plan() / recommend() / ollama_env() are the library entry points; OSR_INFERENCE_* (context, parallel, kv_type, headroom_ratio, bandwidth_gbps, load_gbps, keep_alive ...) set the defaults. Rules of thumb the planner encodes: one warm model per CPU node beats several evicting each other; OLLAMA_KEEP_ALIVE=-1 once the set fits; q8_0 KV halves cache memory with no measurable loss; never let context x parallel push a model out of RAM. tests/test_inference.py::test_curated_azure_set_fits_its_container_apps guards the Azure sizing.

Cold loads are removed twice on the hosted platform: the model servers preload their set at start (OSR_OLLAMA_PRELOAD=1 -> ollama run MODEL "" after the pulls, with OLLAMA_KEEP_ALIVE=-1), and the providers owner (ProviderHealthChecker, OSR_PLATFORM_PROVIDERS_KEEP_WARM_MAX) reads /api/ps on every health pass and loads any mapped model that is missing when the provider maps at most that many distinct models (1 per CPU app, the resident count on a GPU). GET /api/v1/providers exposes providers[].loaded and models[].warm; the console provider card shows "in memory: ...". Test pattern: platform/api/tests/test_model_residency.py (stub Ollama with /v1/models, /api/ps, /api/chat).

The model server (platform/models, osr-models)

The platform's own model host (docs/MODELS.md): Ollama /api/* + OpenAI /v1/* + Anthropic /v1/messages over one llama-server process per loaded model (CPU or CUDA). osr_models/store.py reads and writes Ollama's on-disk layout (blobs/sha256-*, manifests/<registry>/<ns>/<repo>/<tag>), sources.py pulls from ollama.com and Hugging Face (hf.co/<org>/<repo>:<quant> or /<file>.gguf) with resume, digest check and a directory lock shared between replicas, gguf.py reads the header for /api/show and the fit, scheduler.py owns residency (fit against the memory budget, LRU eviction of idle instances only, keep_alive), slots and the bounded queue (Saturated -> 503 + Retry-After), engine.py builds the llama-server command line, api.py is the FastAPI surface, cli.py the osr-models CLI (the image aliases it as ollama, so resources.bicep's start-up script runs unchanged). Settings: OSR_MODELS_* first, the OLLAMA_* names second (settings.py). Image: platform/models/Dockerfile (CPU; BASE_IMAGE/LLAMA_FLAVOUR build args for CUDA) pinned to a llama.cpp release (LLAMA_BUILD, also in azure-pipelines/models.yml). Tests: platform/models/tests (engine-backed ones run when OSR_MODELS_LLAMA_SERVER and OSR_MODELS_TEST_GGUF are set); platform/models/scripts/probe.py <url> is the live end-to-end probe. Deploy: compose llm profile builds it (OSR_MODELS_IMAGE overrides), Azure through OSR_OLLAMA_IMAGE (ollamaImage) on the next Infra run after the Models pipeline pushed <acr>/opensmartroute/models-<env>:ci-<sha>.

Helm

helm install osr deploy/helm/opensmartroute -f values.yaml

values.yaml covers image, replicas, HPA, ingress, service account, a ConfigMap for targets.yaml/rules.yaml, extra env/envFrom for secrets (OSR_MCP_KEY, encrypted-state key), a PVC for OSR_STATE and hardening.* (guard, learnedGuard, redactPii, outputGuard: off|flag|redact|block, health, auditPath, originPolicy, resourceLimits -> the matching OSR_SERVER_* variables, all off by default). With more than one replica use a shared StateStore (Redis/SQL) via a small Python entrypoint, or run one writer and read-only replicas.

Tuning without code

Every tunable is an OSR_<GROUP>_<FIELD> variable - run osr settings to list them with current values. Common ones: OSR_ROUTING_SOFTMAX_TEMPERATURE, OSR_ROUTING_NARROW_ABOVE, OSR_POLICY_JAILBREAK_THRESHOLD, OSR_WEIGHTS_RULES, OSR_WEIGHTS_CAPABILITY, OSR_BANDIT_CONFIDENCE_WINDOW. Bad values fail fast at startup with ConfigurationError.

Operating

• Route latency is sub-millisecond for local strategies; only an LLM judge adds network time - keep escalate_llm_judge_below low and use TimeoutMiddleware.
• Post Outcomes to /feedback from the caller that observed the result; /stats shows them.
• Watch route_latency_ms.p95, policy_rejections_total, errors_by_type from MetricsTelemetry.snapshot(); GET /metrics renders the same store as Prometheus text (MetricsTelemetry.prometheus(namespace="osr", extra=...)). Scrape it with a ServiceMonitor or prometheus.io/scrape annotations. RouterBuilder.with_cache(ttl_s, max_size) adds a decision cache whose hits and misses appear as osr_decision_cache_*; with_audit(sink, outcomes=True) chains outcomes too.
• Tracing: GET /metrics also carries the tracer's osr_spans_total / osr_span_duration_ms per stage, GET /trace/{request_id} returns every span and event of one request (X-OSR-Trace-Id on each response; send traceparent to join your own trace), GET /events?name=route.* filters the buffer. From a terminal: osr trace <request_id> renders that trace as a stage-by-stage timeline (--json for the payload) and osr events --name output_guard.* --level warning --limit 50 lists events, both against the saved osr login credentials (--url / --token / --profile override; work against osr serve and the platform). OSR_OBSERVABILITY_LOG_EVENTS=true for JSON lines, OSR_OBSERVABILITY_EVENTS_FILE=/state/events.jsonl for a file, OSR_OBSERVABILITY_OTEL=true (+ otel extra) for OpenTelemetry, OSR_OBSERVABILITY_SAMPLE_RATE under load. See docs/OBSERVABILITY.md. The hosted platform image reads the same variables; there the buffer is workspace-scoped (GET /api/v1/trace/{request_id} with outcomes and audit records, GET /api/v1/events, public GET /api/v1/status for status pages) - osr-platform skill.
• Use osr -t targets.yaml eval data.jsonl --min-accuracy 0.9 in the image build pipeline so a catalogue change cannot ship a regression.
• Secrets: env or <NAME>_FILE mounts via load_secret; never bake them into the image or YAML.