Answers from the documentation; checking which model is on duty
Hi - I answer from the OpenSmartRoute documentation: routing, the API, plans and quotas, self-hosting. Ask away, or open a support ticket if you need a person.
Grounded in the docs - follow a source before acting on it.
Google releases LiteRT Tensor API for native edge model authoring - OpenSmartRoute
Google releases LiteRT Tensor API for native edge model authoring
Google launched a new API to write AI models directly in C++ for edge devices. This tool allows full control over computation graphs without waiting for framework updates.
Key points
Google released the LiteRT Tensor API for native model authoring on edge devices.
A coding agent increased SAM 2 frame rates from under 10 to over 30 FPS on WebGPU.
The API supports direct weight loading from Hugging Face .safetensors files.
Data stays on the accelerator with zero copies between pre-processing and inference.
Why it matters: Developers gain full control over model graphs and zero-copy data flow for faster edge inference.
By OpenSmartRoute editorial · written through the router by writer-small
Google released the LiteRT Tensor API for native edge model authoring. This tool lets engineers write AI models directly in C++. It runs on mobile devices, laptops, and web browsers.
Google announced this new API to solve edge deployment challenges. Teams can now build custom computation graphs without waiting for framework updates. The release targets devices with tight memory and latency budgets.
What Google announced with the LiteRT Tensor API
Google launched a new lightweight authoring layer within the LiteRT stack. This layer sits between raw C++ code and the final model runtime. It allows developers to define how neural networks connect and operate.
The team designed this to improve the process of handcrafting models. Previously, developers often had to wait for upstream frameworks to release new features. Now, they can optimize the network structure immediately on their specific hardware.
This API combines the flexibility of native C++ with cross-platform portability. Users build the computation graph operation by operation in C++. They choose exactly which mathematical operations run and how they link together.
The same graph can execute on CPU, GPU, or NPU hardware. It also runs directly on CPU using XNNPACK without needing a model file. This means teams do not need to rewrite code for every new device target.
The problem with current native model authoring approaches
Current approaches require developers to write computations manually in native C++ or shader languages. Apple's MLX library made this code-first approach popular on Apple silicon devices. The llama.cpp project showed how tailored C++ implementations can improve LLM inference performance.
However, this level of control comes at a significant cost. Implementations become tightly coupled to specific backends like Metal, OpenCL, or Vulkan. Teams must rewrite the same code for each new hardware target they want to support.
Managing zero-copy buffer interoperability adds substantial manual engineering effort. Engineers must synchronize data across CPU, GPU, and NPU units. They also need to glue together pre-processing, inference, and post-processing steps.
Native audio models process sound directly without converting it to text first. This method preserves timing, tone, and speaker cues that text transcripts lose.
Deepseek released V4.1 Flash 1017 which beats larger rivals in coding and agent tasks. The model also cuts per-task costs significantly while producing shorter outputs.
These challenges make it difficult to support new architectures quickly. Teams often spend more time maintaining the infrastructure than building the model itself.
Key features of the new Tensor API design
The LiteRT Tensor API provides four key features for authoring and optimizing on-device models. First, it offers full control over the graph structure. Developers decide every operation in the model graph and every connection between them.
Second, the API supports direct weight ingestion. Teams can load weights straight from model checkpoints like Hugging Face .safetensors files. They do not need conversion scripts to bridge the gap between formats.
Third, the design includes tensor-centric buffer interop. Pre-processing, model partitions, and post-processing pass tensors to each other through shared accelerator buffers. Intermediate data stays on the accelerator with zero copies.
Fourth, the API provides building blocks for AI agents. Models are plain C++ built from well-defined ops. Coding agents can construct a model, check its results against a reference, and tune its performance by benchmarking variants on each device.
Real-world performance with the Segment Anything model
Google ran the same C++ code for the Segment Anything model on the GPUs of three different devices. They tested it on a macOS laptop running Metal, an iOS phone running Metal, and an Android phone running OpenCL.
In tests on a MacBook Pro, the model ran about 1.5x faster than Apple MLX. This comparison used fp16 precision for the image encoder plus mask decoder. The SAM 2.1 sample includes the benchmark commands so users can reproduce the results.
The team built an interactive Segment Anything App running entirely on-device in the browser on WebGPU. The app enables real-time multi-object segmentation and tracking via Meta's Segment Anything Model 2.
A coding agent wrote and evolved the model from under 10 to over 30 frames per second at 512x512 resolution on WebGPU. The complete implementation and the web app are open source in the LiteRT Samples GitHub repo.
How the API enables zero-copy pipelines and agent tuning
The LiteRT Tensor API enables zero-copy pipelines by keeping data on the accelerator between steps. In a typical app, only the model runs on the GPU while the CPU handles resizing and passing data. Each trip copies data and stalls the GPU.
In the litert-samples/sam2 project, all four steps are Tensor API graphs on the GPU. Pre-processing, the image encoder, the mask decoder, and post-processing are all part of the same graph. The graphs connect once at startup to share buffers, then run for each frame and click.
No data returns to the CPU between steps. All 522 encoder and 269 decoder operations run on the GPU. Pre- and post-processing add only about 0.34 ms per frame. This keeps the whole pipeline fast and efficient.
The API also enables agent tuning because models are plain C++ built from well-defined ops. Coding agents can construct a model, check its results against a reference, and tune its performance. They can benchmark variants on each specific device to find the best configuration.
Why this matters for edge deployment teams
This news matters because it reduces the engineering effort required to deploy models on edge devices. Teams no longer need to maintain separate codebases for different hardware backends. They can write once and run on CPU, GPU, or NPU.
It also matters for speed and cost. Zero-copy pipelines reduce latency by eliminating unnecessary data transfers between CPU and accelerator. Faster inference means better user experiences on mobile devices and web browsers.
Furthermore, the ability to tune graphs for each backend improves quality. Agents can rewrite operations to be more efficient for specific hardware. This leads to higher frame rates and lower power consumption on battery-powered devices.
How OpenSmartRoute helps with native edge models
Teams that route requests through OpenSmartRoute can now easily add native edge models to their catalogue. OpenSmartRoute is an open-source router for AI requests with a hosted platform. It sends each request to the best-fit model, agent, tool, or skill from a catalogue the team defines.
A team can add a new entry for a model authored with the LiteRT Tensor API. This model competes on the next request alongside other options. The router scores every candidate on quality, cost, speed, and safety. The team sets the weights per request to prioritize specific factors.
OpenSmartRoute supports open-weight models served locally on edge devices. This allows teams to keep sensitive data on-premises while using the best performance from the new Tensor API. The router works with any OpenAI-compatible provider and supports MCP tools and A2A agents.
The savings ledger shows what each routed request cost next to what the most expensive model would have cost. This helps teams measure the value of deploying optimized native models. An input guard spots prompt injection and personal data before a request leaves the device.
The 'osr eval' tool measures routing accuracy on the team's own prompts. It can fail a build when the routing accuracy drops below a threshold. This ensures that the new edge models integrate correctly with the existing routing logic.
How it compares
Before this release, teams chose one of two paths for edge AI. They either converted a framework model like PyTorch or wrote custom C++ code. The conversion path is fast. It works for most standard models. But it cannot easily support new architectures. Developers cannot change the lowering logic without waiting for upstream updates.
The custom C++ path gives full control. Teams write every operation in C++ or shader languages. They get maximum performance. But they face heavy costs. Code must be rewritten for every new hardware target. Managing memory buffers across CPU, GPU, and NPU requires substantial manual engineering. Zero-copy buffer interoperability is difficult to achieve without native code.
The new LiteRT Tensor API changes the middle ground. It combines flexibility with portability. Teams build the computation graph op by op in C++ like the custom path. But the graph runs on CPU, GPU, or NPU without rewriting. It also supports running directly on CPU via XNNPACK. This removes the need to rebuild for every device.
The conversion path remains the best default for most models. Developers should start there. They convert a trained model and run it with the LiteRT runtime. This offers the quickest route to production. The Tensor API is for those needing low-level control. It is for trying new lowering logic or supporting new architectures.
What stays the same is the core goal. Shipping on-device AI still means meeting tight latency and memory budgets. The Tensor API does not replace the need for optimization. It simply makes the optimization process easier and more portable.
Questions this leaves open
The source text does not specify the exact performance numbers for the Tensor API on all devices. It only mentions a 1.5x speedup on a MacBook Pro compared to Apple MLX. This comparison is specific to fp16 precision and includes only the image encoder and mask decoder. Pre- and post-processing add about 0.34 ms.
Readers cannot yet know the total inference time for the entire SAM 2 model using the Tensor API. The benchmark commands are included in the sample, but the full results are not explicitly stated in the text. Teams must run the benchmarks themselves to get accurate numbers for their specific hardware.
The text does not explain the licensing terms for the Tensor API itself. It mentions open-source ML Drift powering the GPU. It also notes that the implementation is open source in the LiteRT Samples GitHub repo. However, it does not state if the API is free for commercial use or requires a license.
There is no information on the memory footprint of the Tensor API compared to the conversion path. Teams deploying on devices with limited RAM need to know if this new method uses less memory. The text focuses on zero-copy buffers but does not quantify the memory savings.
The source does not detail how the coding agent evolved the model from 10 to 30 frames per second. It states the agent wrote and evolved the code. It does not explain the specific tuning steps or the number of iterations required. This process remains a black box in the provided text.
Readers must check the LiteRT Samples GitHub repo for the complete implementation details. They should also review the official LiteRT Tensor API guide for setup instructions. The 'osr eval' tool mentioned for OpenSmartRoute helps check routing accuracy but was not tested against the new API in the source text.
The text assumes developers have C++ programming skills. It does not provide a visual guide for beginners. Teams new to native model authoring might find the manual graph construction steep. They need to verify their team's readiness before adopting this approach.
Finally, the text does not mention compatibility with other popular frameworks like TensorFlow or ONNX. It focuses on PyTorch conversion and native C++ authoring. Teams using other frameworks might need to adapt their workflow significantly. They should check the documentation for cross-framework support.
What to do
Start by reading the LiteRT Tensor API guide for getting started and build instructions. The guide explains how to set up the environment and write your first computation graph.
Check the Tensor API reference documentation for details on operations and buffer management. Look for examples of how to load weights from .safetensors files directly into the graph.
Explore the SAM 2 sample repository to see how a coding agent can evolve a model from under 10 to over 30 frames per second. You can try running the live web demo to see the segmentation in action on your own device.
If you are using OpenSmartRoute, update your catalogue with the new native edge models. Define hard rules to pin requests with personal data to on-premises models. Use the savings ledger to track the cost benefits of routing traffic to these optimized models.
Monitor your 'osr eval' results to ensure the new models perform well against your specific prompts. Adjust the weights in your scoring rules if the routing accuracy changes after the update.
