# Model server The OpenSmartRoute model server (`osr-models`) hosts open-weight models on your own hardware - a laptop, a VM, a container, a GPU node - behind the APIs clients already speak. It is what the platform's self-hosted targets run on, and it works on its own for anyone who wants a model server with a fit check, an admission queue and metrics instead of a cold-load lottery. | | | |---|---| | **APIs** | Ollama (`/api/generate`, `/api/chat`, `/api/embed`, `/api/tags`, `/api/show`, `/api/ps`, `/api/pull`, `/api/create`, `/api/copy`, `/api/delete`, `/api/blobs`, `/api/version`), OpenAI (`/v1/models`, `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/responses`, `/v1/rerank`), Anthropic Messages (`/v1/messages`), control (`/models`, `/models/load`, `/models/unload`), operations (`/healthz`, `/readyz`, `/metrics`) | | **Engine** | llama.cpp `llama-server`, one process per loaded model, on CPU or NVIDIA CUDA; GGUF weights | | **Sources** | ollama.com (`qwen3.5:4b`), Hugging Face GGUF repositories (`hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M`, or a file inside a repository), local `.gguf` files | | **Store** | content-addressed blobs and manifests in Ollama's on-disk layout: a store Ollama filled is served as-is, so switching costs no download | | **Residency** | a model is loaded only when its footprint (weights + KV cache x context x slots + overhead) fits the memory budget after evicting *idle* models; a busy model is never evicted; `keep_alive` pins, expires or unloads | | **Admission** | `parallel` slots per model, a bounded queue with a deadline, `503` + `Retry-After` past it (the platform re-routes to a cloud tier on that answer instead of waiting) | | **Metrics** | Prometheus: requests, latency, tokens (prompt / cached / completion), slots, queue, resident bytes, loads, evictions, refusals - plus the engine's own series per model | ## Run it **Docker (CPU)** ```sh docker run -d --name models -p 11434:11434 -v osr-models:/root/.ollama \ -e OSR_OLLAMA_MODELS="qwen3.5:4b qwen3-embedding:4b" -e OSR_OLLAMA_PRELOAD=1 \ opensmartroute/models curl http://localhost:11434/api/chat -d '{"model":"qwen3.5:4b","messages":[{"role":"user","content":"Hello"}],"stream":false}' ``` **Docker (NVIDIA GPU)** - the `cuda` image puts every layer on the GPU (`OSR_MODELS_GPU_LAYERS` to split): ```sh docker run -d --gpus all -p 11434:11434 -v osr-models:/root/.ollama opensmartroute/models:cuda ``` **The command line** - the same binary inside the container manages the store and talks to the server: ```sh docker exec models osr-models pull hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M docker exec models osr-models run hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M "Why is the sky blue?" docker exec models osr-models ps ``` Outside a container the package runs next to any `llama-server` binary from a llama.cpp release (point `OSR_MODELS_LLAMA_SERVER` at it or put it on the `PATH`) and `osr-models serve` starts the server. The server listens on Ollama's port (`11434`) and reads Ollama's variables, so an Ollama client, probe or start-up script works unchanged; the container even answers to `ollama list`, `ollama pull`, `ollama run`. ## Models | Command | What it does | |---|---| | `osr-models pull qwen3.5:4b` | from the ollama.com library (any registry speaking its protocol: `host/namespace/name:tag`) | | `osr-models pull hf.co//:` | the GGUF of that quantisation from a Hugging Face repository (`Q4_K_M` when no quantisation is named); a multimodal projector (`mmproj-*.gguf`) in the same folder comes along; `HF_TOKEN` for gated repositories | | `osr-models pull hf.co///.gguf` | one file of a repository | | `osr-models pull ./model.gguf --name my-model` | import a local file | | `osr-models pull qwen3.5:4b=hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M` | pull from one source, store under another name: every API, `show` and the platform's mapping see `qwen3.5:4b` while the weights come from the Hugging Face repository. Use it where the ollama.com blob is Ollama's own conversion that a standard llama.cpp cannot load (`gpt-oss:20b`, `glm-4.7-flash`, `glm-ocr`, `translategemma:4b`, `qwen3.5:4b`). `POST /api/pull` takes the same form in `model`, or the source in a `source` key | | `osr-models list` / `ps` / `show MODEL` / `rm MODEL` / `stop MODEL` | the store, what is in memory, a model's details and metadata, delete, unload | | `POST /api/create` | a new model from an existing one plus a system prompt, parameters (`temperature`, `stop`, `num_ctx`, ...), a template or adapters - the same body Ollama takes | Pulls resume after an interruption, verify the digest, and are shared between replicas that mount the same store through a lock in it. The deployment decides which sources are allowed (`OSR_MODELS_SOURCES`, default ollama.com and Hugging Face). A model's card is what `POST /api/show` returns: `details` (family, parameter count, quantisation), `model_info` (the GGUF metadata - layer count, heads, trained context), `capabilities` (`completion`, `tools`, `thinking`, `vision`, `embedding`, read off the weights), `system`, `parameters`, `template`, `license`. The system prompt and parameters of a card apply to every request that does not set its own. ## Requests Everything the Ollama and OpenAI dialects define for chat works on both paths: streaming, `tools` and tool calls, images (multimodal models), `format: "json"` or a JSON schema (`response_format` on `/v1`), `think` levels (`reasoning_effort` on `/v1` - `none` switches a thinking model's reasoning off so it answers inside a small `max_tokens`), `keep_alive`, the Ollama `options` (`temperature`, `top_p`, `top_k`, `min_p`, `num_predict`, `seed`, `stop`, penalties). Answers carry Ollama's statistics (`prompt_eval_count`, `prompt_eval_cached_count`, `eval_count`, durations in nanoseconds, `load_duration` on the request that loaded the model) and the OpenAI `usage` with `cached_tokens`. The prompt cache is on: a repeated system prompt or conversation prefix is not evaluated again. An empty request loads a model (`{"model": "...", "messages": []}`), `keep_alive: 0` with it unloads; `GET /api/ps` lists what is resident with its slots, queue and `expires_at` (`0001-01-01` = pinned). ## Settings Every setting is an environment variable; the `OSR_MODELS_*` name wins over the Ollama name it mirrors. | Setting | Ollama name | Default | Meaning | |---|---|---|---| | `OSR_MODELS_HOST` | `OLLAMA_HOST` | `127.0.0.1:11434` | listen address | | `OSR_MODELS_STORE` | `OLLAMA_MODELS` | `~/.ollama/models` if present, else `~/.osr/models` | the store directory (`/root/.ollama/models` in the image) | | `OSR_MODELS_PRELOAD` | `OSR_OLLAMA_MODELS` | - | models pulled at start (`name` or `name=source`; a `name=source` entry whose stored weights came from anywhere else - an earlier ollama.com pull of the same tag - is pulled again from the source); with `OSR_OLLAMA_PRELOAD=1` loaded and pinned too | | `OSR_MODELS_CONTEXT` | `OLLAMA_CONTEXT_LENGTH` | `8192` | context per slot (clamped to the model's trained window and its `num_ctx`) | | `OSR_MODELS_PARALLEL` | `OLLAMA_NUM_PARALLEL` | `1` | generations one model decodes at once (embedding models always 1) | | `OSR_MODELS_KV_CACHE` | `OLLAMA_KV_CACHE_TYPE` | `f16` | KV cache precision (`q8_0` halves it; `q4_0` quarters it) | | `OSR_MODELS_MAX_LOADED` | `OLLAMA_MAX_LOADED_MODELS` | `0` = as many as fit | cap on resident models | | `OSR_MODELS_KEEP_ALIVE` | `OLLAMA_KEEP_ALIVE` | `5m` | idle time before a model is unloaded; `-1` = never | | `OSR_MODELS_QUEUE_MAX` | `OLLAMA_MAX_QUEUE` | `4 x parallel` | requests that may wait for a slot before `503` | | `OSR_MODELS_QUEUE_TIMEOUT` | - | `20s` | longest wait for a slot | | `OSR_MODELS_LOAD_TIMEOUT` | `OLLAMA_LOAD_TIMEOUT` | `20m` | longest load (a network share is slow) | | `OSR_MODELS_REQUEST_TIMEOUT` | - | `15m` | longest silence from an engine during one request (a thinking model on a CPU can be quiet for minutes) | | `OSR_MODELS_DEVICE` | - | `auto` | `cpu`, `cuda`, or detect | | `OSR_MODELS_GPU_LAYERS` | - | all | layers offloaded to the GPU | | `OSR_MODELS_THREADS` | - | engine default | CPU threads per engine | | `OSR_MODELS_FLASH_ATTENTION` | `OLLAMA_FLASH_ATTENTION` | on | flash attention (forced on for a quantised KV cache) | | `OSR_MODELS_MEMORY` | - | detected (cgroup limit or physical memory) | the memory budget before headroom, e.g. `28Gi` | | `OSR_MODELS_HEADROOM` | - | `0.12` | share of memory kept for the OS and page cache | | `OSR_MODELS_SOURCES` | - | `registry.ollama.ai huggingface.co hf.co` | registries pulls may use | | `OSR_MODELS_HF_TOKEN` | `HF_TOKEN` | - | Hugging Face token for gated repositories | | `OSR_MODELS_TOKEN` | - | - | bearer token required off the loopback (`/healthz`, `/readyz`, `/api/version` stay open) | | `OSR_MODELS_SLEEP_IDLE` | - | off | seconds of idleness after which an engine releases its memory but keeps its process | | `OSR_MODELS_LLAMA_SERVER` | - | `PATH` | the engine binary | Sizing is arithmetic the SDK does for you: `osr capacity show --memory 28Gi --cores 4` prints a model's footprint, whether it fits and the expected tokens/s; `osr capacity plan --memory 28Gi` packs a set onto hosts; `osr capacity ps --url http://:11434` lists what the server holds right now. ## With the platform The platform treats a model server as an `ollama` provider: its base URL (`http://:11434/v1`) in `OSR_PLATFORM_PROVIDERS` or the operator console, targets mapped to model names, `concurrency` equal to the slots the server offers. The provider check reads `/api/ps` for residency, the keep-warm pass loads mapped models that are not resident, and the calibrator turns the server's measured latency into what the router ranks on. The Docker Compose stacks run the model server as the `llm` profile; the Azure template's self-hosted model apps run it when `OSR_OLLAMA_IMAGE` names the model server image (the Ollama image stays the default) - same volume, same start-up, same probes. ## Operations - `GET /healthz` - engine version, device, instances, memory budget and use, store usage, preload progress and errors; `GET /readyz` - `200` once the preload list is pulled (and loaded when asked), `503` before. - `GET /metrics` - `osr_models_requests_total`, `osr_models_request_seconds`, `osr_models_tokens_total`, `osr_models_slots_busy`, `osr_models_queue`, `osr_models_resident_bytes`, `osr_models_loads_total`, `osr_models_evictions_total`, `osr_models_refused_total`, `osr_models_load_seconds`, and the engine's `llamacpp:*` series labelled by model. - `GET /models` - every model in the store with its status (`unloaded`, `loading`, `loaded`), footprint and the engine arguments; `GET /models//logs` - the engine's last log lines when a load fails. - A load that does not fit answers `507` with the budget and what to change (shorter context, `q8_0` KV cache, a smaller quantisation). A saturated model answers `503` with `Retry-After`. - Prompts and answers are never written to disk; the engine processes bind to the loopback only.