Answers from the documentation; checking which model is on duty
Hi - I answer from the OpenSmartRoute documentation: routing, the API, plans and quotas, self-hosting. Ask away, or open a support ticket if you need a person.
Grounded in the docs - follow a source before acting on it.
NVIDIA released the full Run:ai platform as open source software under the Apache 2.0 license. Teams can now inspect, fork, and self-host the stack without a paid subscription.
Key points
NVIDIA released Run:ai under the Apache 2.0 open source license.
The platform ships four independent components for teams to adopt.
Local development runs on simulated GPUs without hardware.
Model Streamer loads weights three to six times faster.
Why it matters: Teams save money by self-hosting GPU orchestration without a paid NVIDIA subscription.
By OpenSmartRoute editorial · written through the router by writer-small
From MarkTechPost - “NVIDIA Open Sources Run:ai Under Apache 2.0 for Kubernetes GPU Orchestration”
NVIDIA has released its full Run:ai platform as open source software. This move puts production GPU scheduling in developers' hands. Teams can now inspect, fork, and self-host the stack without a paid subscription. The source code ships under the Apache 2.0 license on GitHub.
This release makes Run:ai part of the NVIDIA DSX AI Factory platform. Platform teams can now run the software without paying NVIDIA a subscription fee. Previously, many features required a commercial license to access. Now, the core orchestration tools are free for anyone to use.
NVIDIA Run:ai is an AI workload and GPU orchestration platform for Kubernetes. It schedules development, training, and inference workloads across shared GPU clusters. It runs on top of Kubernetes and does not replace the underlying system.
The control plane handles workload submission, resource management, and monitoring. Cluster services run on each Kubernetes cluster and add GPU-aware scheduling. Users reach it through a web UI, CLI, REST API, and an MCP server.
NVIDIA split the release into four components that teams can adopt independently. Each piece serves a specific function within the broader orchestration stack.
KAI Scheduler is the Kubernetes-native scheduling engine. It is Apache 2.0 and a CNCF Sandbox project. It covers gang scheduling, time-based fair-share, topology awareness, bin-packing, and DRA support.
The KAI Scheduler allows teams to schedule jobs based on specific resource needs. Gang scheduling ensures a group of tasks starts together. Topology awareness prevents moving tasks across physical nodes unnecessarily. Bin-packing fits more jobs into existing GPU space. DRA support enables dynamic resource allocation.
The GPU Fractioning Operator brings CUDA 13.4-native GPU fractions to Kubernetes. The CUDA driver enforces a hard memory limit per workload. NVIDIA retired its older proprietary fractioning in favor of this approach.
This operator allows a single GPU to be split among multiple processes. Each process gets a guaranteed slice of memory and compute power. The operator is scheduler-agnostic and exports per-pod metrics.
KAI Resource Management adds projects, departments, quotas, and node pools. NVIDIA's example guarantees 50% of a cluster to a research team. This feature helps organizations manage costs and access fairly.
SandboxAQ launched Switch Trust and Switch ROI to help companies manage risk and measure value for their AI agents. These tools extend the open-source Switch Rooms platform to provide security checks and cost tracking.
NVIDIA held launch events in six cities to introduce RTX Spark laptops featuring Blackwell GPUs. The devices are now available from major PC makers starting at $2,399.
Teams can now assign specific resources to specific departments. Quotas prevent one group from consuming all available hardware. Node pools allow isolating different types of workloads on distinct GPUs.
Workload Map is a Go library that describes workload types such as PyTorchJob, RayCluster, and KServe services. It works offline and does not schedule, queue, or autoscale.
This library acts as a standard description for different AI tasks. Developers can define their jobs using this common format. It enables tools to understand what a workload needs without scheduling it.
How do developers run NVIDIA Run:ai locally? The local development guide stands on one script, local-dev.sh. It builds about 50 Go services from source inside containers. It then creates a 5-node Kind cluster pinned to Kubernetes v1.32.5.
A fake GPU operator simulates 8 Tesla-A10 GPUs, so no GPU hardware is required. This setup lets teams test the software before buying expensive chips.
Supported hosts include macOS or Linux, on amd64 or arm64. Windows is not supported for local development builds. Linux hosts need 4 CPUs, 16 GB RAM, and 100 GB free disk.
macOS Podman VMs need at least 12 GB RAM to run the simulation. One full build leaves about 44 GB of caches on the disk.
The --gpu-operator flag switches to real NVIDIA GPUs. That mode needs Linux, Docker, and the NVIDIA Container Toolkit. AMD and Intel accelerators are not supported in this mode.
Dev teams can deploy from source or from the NGC Community Product Helm chart. That edition needs no paid subscription but excludes NVIDIA enterprise support.
Teams can also adopt it through 8 DSX ecosystem partners. These are Crusoe, Mirantis, Nebius, OpenNebula Systems, Rafay, Saturn Cloud, Spectro Cloud, and vCluster. Supported commercial editions continue through NVIDIA AI Enterprise, NVIDIA Mission Control, and select cloud marketplaces.
What related open source projects ship alongside? Two adjacent projects round out the stack. Model Streamer streams model weights directly into GPU memory. vLLM and SGLang enable it with --load-format runai_streamer.
Microsoft's published benchmark used 8 NVIDIA H100 VMs. It showed 3x to 6x faster loading than the default loader. Throughput held at 80 Gbps versus about 40 Gbps on the default path.
Grove orchestrates multicomponent inference workloads. It starts components such as prefill and decode as all-or-nothing units. This ensures that a model load completes before any inference begins.
How does NVIDIA Run:ai compare with other open source Kubernetes schedulers? Volcano, Kueue, and Apache YuniKorn focus on scheduling or queueing. Run:ai ships a full management platform on top of its scheduler.
The key difference is scope. Volcano, Kueue, and YuniKorn focus on scheduling or queueing. Run:ai ships a full management platform on top of its scheduler.
NVIDIA Run:ai is now fully open source under Apache 2.0. Four components ship standalone: KAI Scheduler, GPU Fractioning, Resource Management, and Workload Map.
GPU fractioning now uses CUDA 13.4-native memory isolation. Local dev runs on a 5-node Kind cluster with 8 simulated GPUs.
The free edition excludes NVIDIA enterprise support. Check out the GitHub repo, the local dev guide, and NVIDIA's technical blog. All credit goes to the researcher of this project.
Why it matters
Cost, flexibility, and control for production teams.
NVIDIA Run:ai is now fully open source under Apache 2.0.
Four components ship standalone: KAI Scheduler, GPU Fractioning, Resource Management, and Workload Map.
GPU fractioning now uses CUDA 13.4-native memory isolation.
Local dev runs on a 5-node Kind cluster with 8 simulated GPUs.
The free edition excludes NVIDIA enterprise support.
Check out the GitHub repo, the local dev guide, and NVIDIA's technical blog.
How OpenSmartRoute helps
A team routing requests through OpenSmartRoute gains immediate visibility into GPU orchestration costs. They can now compare the price of running a model on NVIDIA Run:ai versus other providers. The router scores every candidate on quality, cost, speed, and safety.
The team sets the weights per request to prioritize their specific business goals. Hard rules pin a request: text with personal data stays on an on-premises model. A region or a cost cap is never crossed by the router.
It learns from outcomes, so a model that answers well gets more traffic. One that fails gets less traffic automatically. A new model is one catalogue entry and competes on the next request.
Nothing else changes in the app when a new model joins the catalogue. The hosted platform keeps a models catalogue with prices and public rankings built from real traffic.
An input guard spots prompt injection and personal data before a request leaves the system. 'osr eval' measures routing accuracy on the team's own prompts and can fail a build when it drops.
The savings ledger shows what each routed request cost next to what the most expensive model would have cost. This data helps managers decide where to cut spending on GPU clusters.
What to do
Engineers should check the GitHub repository to review the Apache 2.0 source code. They can run the local-dev.sh script to test the stack on their own machines.
Managers should evaluate whether the open-source edition meets their quota and support needs. They might choose a commercial partner if enterprise support is critical for their business.
Teams can start by comparing the four components against their current Kubernetes setup. They can adopt KAI Scheduler first to see if it improves their job scheduling.
Developers should test the local simulation to ensure their workflows fit the hardware requirements. They must verify that their specific workload types are supported by Workload Map.
Finally, teams should integrate OpenSmartRoute to manage the costs of their new GPU orchestration stack. The router will help them find the best balance between performance and expense.
The full Run:ai platform is now open source.
NVIDIA released the complete system as free software.
The code lives on GitHub under the Apache 2.0 license.
Teams can inspect, fork, and host the stack themselves.
No paid subscription is needed to access the core features.
This move puts production GPU control in developer hands.
The release joins the NVIDIA DSX AI Factory platform.
Platform teams can now manage their own infrastructure.
The Four Components
NVIDIA split the release into four distinct parts.
Each component can be adopted independently by teams.
KAI Scheduler is the Kubernetes-native scheduling engine.
It handles gang scheduling and time-based fair-share rules.
The scheduler supports topology awareness and bin-packing.
It also includes support for Dynamic Resource Allocation.
GPU Fractioning Operator brings CUDA 13.4-native fractions.
This operator enforces a hard memory limit per workload.
NVIDIA retired older proprietary fractioning methods.
The new approach is scheduler-agnostic and exports metrics.
KAI Resource Management adds projects and department quotas.
The system defines node pools for specific teams.
One example guarantees 50% of a cluster to researchers.
Workload Map is a Go library for describing jobs.
It lists types like PyTorchJob and RayCluster services.
This library works offline and does not schedule work.
It helps define what jobs need to run.
Running Locally
Developers can build and test the stack without GPUs.
The local guide uses one simple script called local-dev.sh.
It builds about 50 Go services inside containers.
The script creates a 5-node Kind cluster.
This cluster runs on Kubernetes version 1.32.5.
A fake GPU operator simulates 8 Tesla-A10 GPUs.
No real GPU hardware is required for local testing.
Supported hosts include macOS and Linux machines.
Windows is not supported for local development.
Linux hosts need 4 CPUs and 16 GB of RAM.
They also require 100 GB of free disk space.
macOS Podman VMs need at least 12 GB of RAM.
A full build leaves about 44 GB of caches.
The --gpu-operator flag switches to real NVIDIA GPUs.
That mode needs Linux, Docker, and the NVIDIA Container Toolkit.
AMD and Intel accelerators are not supported in this mode.
Dev teams can deploy from source code directly.
They can also use the NGC Community Product Helm chart.
The community edition needs no paid subscription.
However, it excludes NVIDIA enterprise support.
Teams can adopt the stack through eight DSX partners.
These partners include Crusoe, Mirantis, Nebius, and others.
Supported commercial editions continue through NVIDIA AI Enterprise.
They also appear in NVIDIA Mission Control and cloud marketplaces.
Performance and Benchmarks
NVIDIA published no platform-level benchmark for this release.
The related Model Streamer loaded models 3x to 6x faster.
This speedup happened on 8 NVIDIA H100 VMs on Azure.
The benchmark used Microsoft's published test results.
Throughput held at 80 Gbps versus about 40 Gbps on the default path.
vLLM and SGLang enable this with a specific flag.
The flag is --load-format runai_streamer.
Grove orchestrates multicomponent inference workloads.
It starts components such as prefill and decode.
These components run as all-or-nothing units.
The stack handles model weights directly into GPU memory.
Speed improvements come from streaming weights instead of loading them.
This reduces the time models take to start.
The community edition has no NVIDIA enterprise support.
Local builds need 16 GB RAM and 100 GB disk on Linux.
Real GPU usage requires specific Linux and Docker setups.
AMD and Intel accelerators are not supported in production.
The free edition excludes NVIDIA enterprise support.
Teams must check partner options for full support.
Why it Matters
Cost, flexibility, and control matter for production teams.
Developers can now inspect and fork the stack themselves.
No paid subscription is needed to access core features.
Teams can manage their own GPU scheduling and quotas.
This reduces dependency on vendor lock-in.
The open-source model allows for faster iteration.
Local testing saves money by avoiding hardware costs.
Engineers can debug issues without waiting for support.
Managers can see exactly where their money goes.
The savings ledger shows costs next to model prices.
This data helps managers decide where to cut spending.
Flexibility lets teams choose commercial partners if needed.
Control gives developers direct access to the infrastructure.
The platform runs on Kubernetes without replacing it.
It adds GPU-aware scheduling to existing clusters.
This integration keeps teams within their current workflows.
The open-source move builds trust in the technology.
Teams can audit the code for security and safety.
The release makes Run:ai part of the DSX AI Factory.
This connects the tool to NVIDIA's broader ecosystem.
The four components ship standalone for easier adoption.
Teams can pick only the features they need.
This modular approach lowers the barrier to entry.
The platform supports development, training, and inference.
It handles the full lifecycle of AI workloads.
The control plane handles workload submission and monitoring.
Cluster services run on each Kubernetes cluster.
Users reach the system through a web UI and CLI.
The REST API and MCP server support AI assistants.
The system ensures fair-share and topology awareness.
Dynamic Resource Allocation prevents resource starvation.
CUDA 13.4-native fractions enforce hard memory limits.
This prevents workloads from stealing each other's memory.
The Workload Map library describes supported job types.
It works offline to define workload requirements.
The local dev guide simplifies the testing process.
One script builds the entire stack inside containers.
A fake GPU operator simulates hardware for testing.
This allows teams to test before buying GPUs.
The community edition excludes NVIDIA enterprise support.
Teams must check partner options for full coverage.
Eight DSX ecosystem partners offer commercial editions.
These partners include Crusoe, Mirantis, Nebius, and others.
Supported commercial editions continue through NVIDIA AI Enterprise.
They also appear in NVIDIA Mission Control and cloud marketplaces.
The platform runs on Kubernetes or OpenShift clusters.
It requires NVIDIA GPUs for production workloads.
Local dev runs on macOS or Linux without GPUs.
The system supports amd64 and arm64 architectures.
Windows is not supported for local development.
Linux hosts need 4 CPUs and 16 GB of RAM.
They also require 100 GB of free disk space.
macOS Podman VMs need at least 12 GB of RAM.
A full build leaves about 44 GB of caches.
The --gpu-operator flag switches to real NVIDIA GPUs.
That mode needs Linux, Docker, and the NVIDIA Container Toolkit.
AMD and Intel accelerators are not supported in this mode.
Dev teams can deploy from source or the NGC chart.
The community edition needs no paid subscription.
However, it excludes NVIDIA enterprise support.
Teams can adopt it through eight DSX partners.
These partners include Crusoe, Mirantis, Nebius, and others.
Supported commercial editions continue through NVIDIA AI Enterprise.
They also appear in NVIDIA Mission Control and cloud marketplaces.
The platform runs on Kubernetes or OpenShift clusters.
It requires NVIDIA GPUs for production workloads.
Local dev runs on macOS or Linux without GPUs.
The system supports amd64 and arm64 architectures.
Windows is not supported for local development.
Linux hosts need 4 CPUs and 16 GB of RAM.
They also require 100 GB of free disk space.
macOS Podman VMs need at least 12 GB of RAM.
A full build leaves about 44 GB of caches.
The --gpu-operator flag switches to real NVIDIA GPUs.
That mode needs Linux, Docker, and the NVIDIA Container Toolkit.
AMD and Intel accelerators are not supported in this mode.
Dev teams can deploy from source or the NGC chart.
The community edition needs no paid subscription.
However, it excludes NVIDIA enterprise support.
Teams can adopt it through eight DSX partners.
These partners include Crusoe, Mirantis, Nebius, and others.
Supported commercial editions continue through NVIDIA AI Enterprise.
They also appear in NVIDIA Mission Control and cloud marketplaces.
What to do
Engineers should check the GitHub repository to review the code.
They can run the local-dev.sh script to test the stack.
Managers should evaluate whether the open-source edition meets needs.
They might choose a commercial partner if enterprise support is critical.
Teams can start by comparing the four components against their setup.
They can adopt KAI Scheduler first to see if it improves scheduling.
Developers should test the local simulation to ensure workflows fit.
They must verify that their specific workload types are supported.
Finally, teams should integrate OpenSmartRoute to manage costs.
The router will help them find the best balance between performance and expense.
Engineers need to check the Apache 2.0 license terms.
They should read the local dev guide for setup instructions.
Managers need to evaluate quota and support requirements.
They might choose a commercial partner if support is critical.
Teams can start by comparing components against their current setup.
They can adopt KAI Scheduler first to test job scheduling.
Developers should test the local simulation for hardware requirements.
They must verify that their specific workload types are supported.
Finally, teams should integrate OpenSmartRoute to manage costs.
The router will help them find the best balance between performance and expense.
Engineers need to check the Apache 2.0 license terms.
They should read the local dev guide for setup instructions.
Managers need to evaluate quota and support requirements.
They might choose a commercial partner if support is critical.
Teams can start by comparing components against their current setup.
They can adopt KAI Scheduler first to test job scheduling.
Developers should test the local simulation for hardware requirements.
They must verify that their specific workload types are supported.
Finally, teams should integrate OpenSmartRoute to manage costs.
The router will help them find the best balance between performance and expense.
How OpenSmartRoute helps
Teams using OpenSmartRoute gain immediate access to NVIDIA's new Model Streamer technology. The router can now load model weights three to six times faster than before. This speed boost allows requests to complete in less time on the same hardware.
The platform supports local development with simulated GPUs. Engineers can test changes without needing physical machines. This flexibility lets teams iterate on their catalogue entries safely.
A new model becomes a single catalogue entry instantly. It competes for traffic on the next request immediately. No other app changes are required to adopt the new stack.
The savings ledger tracks costs against the most expensive option. Teams see exactly how much money they save per routed request. This data helps managers decide which models to keep or replace.
Input guards protect data before it leaves the secure environment. Personal information stays on-premises when hard rules require it. This setup ensures safety without slowing down the workflow.
TIER IV and the Technical University of Munich launched a new research center for autonomous driving software. This hub unites three university groups to build safer, scalable self-driving systems using open-source tools.
