The OpenSmartRoute compatible proxy provides a standardized URL – http://router:8000/v1 – for interacting with the official OpenAI SDKs. This allows clients to point directly to the proxy, simplifying integration and providing a consistent interface regardless of the underlying LLM provider.
This proxy utilizes building blocks such as execution.execute_with_fallback for robust routing and error handling, execution.ChatStream for streaming responses, and server.proxy_request / proxy_response for efficient data conversion. The proxy is tested against the official OpenAI SDK and MCP SDK, ensuring compatibility and reliable performance.
The OpenAICompatClient adapter provides a standard client for interacting with OpenAI, Azure OpenAI, vLLM, Ollama, LiteLLM, OpenRouter, Groq and Mistral. It handles retries and provides a consistent API. The proxy is a drop-in base URL for the official OpenAI SDKs and the MCP endpoint for the official MCP clients. The osr serve process builds a plain Router from the catalogue; the production controls are one flag (or one OSR_SERVER_* variable, for the image and the Helm chart) each and wrap it in an EnterpriseRouter through enterprise.harden().
Key Features:
- Standardized URL:
http://router:8000/v1simplifies client integration. - Built-in Fallbacks:
execution.execute_with_fallbackhandles routing failures. - Streaming Support:
execution.ChatStreamenables real-time response processing. - Multi-Provider Compatibility: Supports OpenAI, Azure OpenAI, vLLM, Ollama, LiteLLM, OpenRouter, Groq and Mistral.
Example Configuration:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1")
client.chat.completions.create(model="osr/auto", messages=[{"role": "user", "content": "..."}])
Tools and Capabilities:
The proxy supports deterministic calculator tools and image generation tools, mapping them to compatible providers. The calculator tool answers questions deterministically without a provider, while image tools generate pictures as Markdown, counting as one completion token.
Next Steps:
