AI4 min read
Self-hosted LLM Gateway: Open Source Model Serving Guide
Deploy a self-hosted LLM gateway with open-source model serving. Manage inference pools, gateways, and controllers for private AI deployments.
From growth-engine
Loading
90 days of the enterprise, pro and team plans free, no card needed. See the plans
Blog
Daily notes on new models, LLM releases, agent frameworks and AI research, written from the sources we follow and delivered as a newsletter every day.
Get the morning and evening issues
Every new post of the day, in one email. Confirmation required.
AI4 min read
Deploy a self-hosted LLM gateway with open-source model serving. Manage inference pools, gateways, and controllers for private AI deployments.
From growth-engine
LLMs8 min read
A test showed Qwen3.8-27B computes sums and returns the answer in English words. It succeeded when reasoning was enabled but failed without it.
From Simon Willison on LLMs
Photo: Yan Krukau
Learn how an LLM router directs queries to suitable models, optimizing performance and cost. Explore open-source options and implementation guidance.
From growth-engine
LLMs1 min read
You’re deploying a model on a system. It starts up, prompts are getting responses. Now the hard question: Is this fast? Your instincts might lead you to send...
From NVIDIA technical blog
AI1 min read
A guide ranks 11 open-source agent harnesses based on local inference documentation, licenses, and safety controls.
From MarkTechPost
LLMs1 min read
Researchers propose inference networks that route queries to lower-cost models first, activating expensive experts only when confidence drops below a specific threshold.
From arXiv cs.CL
LLMs1 min read
arXiv:2609.13682v1 Announce Type: new Abstract: We introduce LayerRoute, a parameter-efficient method for adaptive transformer layer-skipping that combines per-layer hard-gated routing (trained via a straight-through estimator) with join...
From arXiv cs.CL
LLMs1 min read
X-CoSD is a new distributed inference framework for large language models that reduces communication load by handling heterogeneous vocabularies. Experiments show improved token generation speed with comparable generation quality to the server LLM.
From arXiv cs.CL
Agents1 min read
This post details deploying the 2.4-trillion parameter Qwen3.8-2.4T-A95B model on Amazon SageMaker HyperPod using vLLM. The setup includes NVFP4 quantization and an OpenAI-compatible endpoint with reasoning and tool calling capabilities.
From AWS machine learning blog
Agents1 min read
This benchmark compares the performance of Qwen3-Coder-30B and NVIDIA Nemotron-3-Nano-30B across G5, G6, G6e, and G7 GPU instances on SageMaker AI. G7 instances demonstrate price-performance gains for real-time LLM inference.
From AWS machine learning blog
LLMs1 min read
NVIDIA's blog discusses using speculative decoding to accelerate large language model inference while preserving accuracy, part of an AI model co-design series.
From NVIDIA technical blog
LLMs1 min read
NVIDIA Dynamo introduces Shadow Engine Recovery, allowing LLM inference engine processes to recover in seconds instead of minutes. This reduces downtime and improves operational efficiency for production deployments.
From NVIDIA technical blog
Posts are drafted from public feeds by models OpenSmartRoute routes to - the same router, skill and metering customers use - and always link to the original source. Corrections: support.
Archive (100)
