Model server

The OpenSmartRoute model server (osr-models) hosts open-weight models on your own hardware - a laptop, a VM, a container, a GPU node - behind the APIs clients already speak. It is what the platform's self-hosted targets run on, and it works on its own for anyone who wants a model server with a fit check, an admission queue and metrics instead of a cold-load lottery.

Run it

Docker (CPU)

docker run -d --name models -p 11434:11434 -v osr-models:/root/.ollama \
  -e OSR_OLLAMA_MODELS="qwen3.5:4b qwen3-embedding:4b" -e OSR_OLLAMA_PRELOAD=1 \
  opensmartroute/models
curl http://localhost:11434/api/chat -d '{"model":"qwen3.5:4b","messages":[{"role":"user","content":"Hello"}],"stream":false}'

Docker (NVIDIA GPU) - the cuda image puts every layer on the GPU (OSR_MODELS_GPU_LAYERS to split):

docker run -d --gpus all -p 11434:11434 -v osr-models:/root/.ollama opensmartroute/models:cuda

The command line - the same binary inside the container manages the store and talks to the server:

docker exec models osr-models pull hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
docker exec models osr-models run hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M "Why is the sky blue?"
docker exec models osr-models ps

Outside a container the package runs next to any llama-server binary from a llama.cpp release (point OSR_MODELS_LLAMA_SERVER at it or put it on the PATH) and osr-models serve starts the server.

The server listens on Ollama's port (11434) and reads Ollama's variables, so an Ollama client, probe or start-up script works unchanged; the container even answers to ollama list, ollama pull, ollama run.

Models

Pulls resume after an interruption, verify the digest, and are shared between replicas that mount the same store through a lock in it. The deployment decides which sources are allowed (OSR_MODELS_SOURCES, default ollama.com and Hugging Face).

A model's card is what POST /api/show returns: details (family, parameter count, quantisation), model_info (the GGUF metadata - layer count, heads, trained context), capabilities (completion, tools, thinking, vision, embedding, read off the weights), system, parameters, template, license. The system prompt and parameters of a card apply to every request that does not set its own.

Requests

Everything the Ollama and OpenAI dialects define for chat works on both paths: streaming, tools and tool calls, images (multimodal models), format: "json" or a JSON schema (response_format on /v1), think levels (reasoning_effort on /v1 - none switches a thinking model's reasoning off so it answers inside a small max_tokens), keep_alive, the Ollama options (temperature, top_p, top_k, min_p, num_predict, seed, stop, penalties). Answers carry Ollama's statistics (prompt_eval_count, prompt_eval_cached_count, eval_count, durations in nanoseconds, load_duration on the request that loaded the model) and the OpenAI usage with cached_tokens. The prompt cache is on: a repeated system prompt or conversation prefix is not evaluated again.

An empty request loads a model ({"model": "...", "messages": []}), keep_alive: 0 with it unloads; GET /api/ps lists what is resident with its slots, queue and expires_at (0001-01-01 = pinned).

Settings

Every setting is an environment variable; the OSR_MODELS_* name wins over the Ollama name it mirrors.

Sizing is arithmetic the SDK does for you: osr capacity show <model> --memory 28Gi --cores 4 prints a model's footprint, whether it fits and the expected tokens/s; osr capacity plan <models...> --memory 28Gi packs a set onto hosts; osr capacity ps --url http://<server>:11434 lists what the server holds right now.

With the platform

The platform treats a model server as an ollama provider: its base URL (http://<server>:11434/v1) in OSR_PLATFORM_PROVIDERS or the operator console, targets mapped to model names, concurrency equal to the slots the server offers. The provider check reads /api/ps for residency, the keep-warm pass loads mapped models that are not resident, and the calibrator turns the server's measured latency into what the router ranks on. The Docker Compose stacks run the model server as the llm profile; the Azure template's self-hosted model apps run it when OSR_OLLAMA_IMAGE names the model server image (the Ollama image stays the default) - same volume, same start-up, same probes.

Operations

• GET /healthz - engine version, device, instances, memory budget and use, store usage, preload progress and errors; GET /readyz - 200 once the preload list is pulled (and loaded when asked), 503 before.
• GET /metrics - osr_models_requests_total, osr_models_request_seconds, osr_models_tokens_total, osr_models_slots_busy, osr_models_queue, osr_models_resident_bytes, osr_models_loads_total, osr_models_evictions_total, osr_models_refused_total, osr_models_load_seconds, and the engine's llamacpp:* series labelled by model.
• GET /models - every model in the store with its status (unloaded, loading, loaded), footprint and the engine arguments; GET /models/<id>/logs - the engine's last log lines when a load fails.
• A load that does not fit answers 507 with the budget and what to change (shorter context, q8_0 KV cache, a smaller quantisation). A saturated model answers 503 with Retry-After.
• Prompts and answers are never written to disk; the engine processes bind to the loopback only.