Google Opens ML Drift for On-Device GPU Inference

Performance benchmarks - memory and latency results

Google benchmarked ML Drift using Gemma models. Gemma is a family of open-weight models. The benchmarks ran on developer-grade workstation hardware. ML Drift showed a 12% lower memory overhead. Memory overhead is the extra memory used by the engine. Other frameworks include llama.cpp and MLX LM. These are other tools for running models. Google emphasized that ML Drift is already running on millions of devices. It powers apps like Chrome, YouTube Shorts, Photos, and Meet. These apps use ML Drift for on-device tasks. In YouTube Shorts, ML Drift achieves up to a 40% reduction in average frame latency. Frame latency is the time it takes to render one frame. This happens for segmentation-based effects on Android and iOS. Segmentation separates objects from the background. In Google Photos, ML Drift speeds up computational photography and segmentation pipelines by up to two seconds. Computational photography uses AI to improve photos. Adobe Lightroom and Photoshop benefit from performance improvements of up to 30%. This applies to AI-powered photo-editing features. Examples include Select Subject, Select Sky, and Adaptive Portrait. Snapchat sees a 30% reduction in model latency for ML-powered face and style-generation effects on Android. This enables the use of larger diffusion models. Diffusion models generate images from noise. Pavithra Agraharam, an AI/ML engineer, noted that native 5D tensor support without layout hacks is a win. Layout hacks are workarounds to fit data into memory. Decoupling logical tensors from physical memory allocations cuts down on memory fragmentation. Matthew Charles Vladislav Busel, founder of Tensorust, asked how ML Drift compares with Core ML on recent iPhones. Core ML is Apple's framework for on-device ML. Google has not provided comparative figures for this.

Running on real devices - use cases and apps

ML Drift runs on millions of devices through popular apps. Chrome uses it for graphics tasks. YouTube Shorts uses it for segmentation effects. Google Photos uses it for computational photography. Google Meet uses it for on-device processing. The engine targets edge devices. These devices have limited power and memory. ML Drift helps these devices run complex models. It supports traditional computer vision. This includes tasks like object detection and image classification. It also supports audio processing. This includes speech recognition and sound effects. Depth processing is another use case. This involves measuring distance to objects. Generative AI is a key target. ML Drift enables larger diffusion models on mobile devices. Diffusion models are powerful but resource-heavy. The engine's optimizations make them feasible on phones. Developers can use the LiteRT ML Drift accelerator. This is a high-level API for application developers. GPU and system engineers can access the lower level C++ API. This API lets them construct custom GpuModel compute graphs. They can also write custom shaders. Shaders are small programs that run on the GPU.

Why it matters - portability and performance

ML Drift matters because it solves a portability problem. Developers write code once. They want it to run on many devices. ML Drift abstracts the underlying GPU APIs. This means the same code can run on OpenGL ES, OpenCL, Metal, and WebGPU. This reduces the effort to support new hardware. It also matters for performance. The benchmarks show real gains. A 12% lower memory overhead saves battery life. A 40% lower frame latency makes animations smoother. A 30% faster photo edit feels snappier. These improvements come from hardware-specific optimizations. ML Drift targets Arm, Intel, and Qualcomm GPUs. It maximizes hardware utilization. This means the GPU works harder and more efficiently. The engine's tensor virtualization and 5D support help spatial and transformer models. These models are the backbone of modern AI. By open-sourcing the engine, Google gives the community a choice. Teams can build custom runtimes. They can integrate ML Drift into their own projects. This fosters innovation in the edge AI space. The move also signals a shift in strategy. Google is focusing on a new foundation for on-device inference. It is phasing out the legacy TensorFlow Lite GPU delegate. This forces teams to plan for the future.

How OpenSmartRoute helps - routing inference to the right backend

Teams that route AI requests through OpenSmartRoute gain flexibility. They can add ML Drift to their catalogue of backends. OpenSmartRoute sends each request to the best-fit model, agent, tool, or skill. The team defines the catalogue. The router scores every candidate on quality, cost, speed, and safety. The team sets the weights for each request. For example, a request with personal data stays on an on-premises model. A region or cost cap is never crossed. The router learns from outcomes. A model that answers well gets more traffic. A model that fails gets less. A new model is one catalogue entry. It competes on the next request. Nothing else changes in the app. This applies to ML Drift as well. The router can send inference tasks to ML Drift on edge devices. It can also send tasks to cloud GPUs. The router balances the load. It respects cost and safety rules. The savings ledger shows what each routed request cost. It compares this to what the most expensive model would have cost. This helps teams track the financial impact of using ML Drift. An input guard spots prompt injection and personal data before a request leaves. This keeps data safe. The hosted platform keeps a models catalogue with prices and public rankings. These rankings come from real traffic. This data helps the router make better decisions about where to send requests.

What to do - migration steps for developers

Teams should migrate from the legacy TensorFlow Lite GPU delegate to the LiteRT ML Drift accelerator. Google will stop updating the legacy delegate. The company recommends this change for backward compatibility with existing models. Application developers can use the LiteRT ML Drift accelerator. They do not need to touch the low-level C++ API. They can simply switch the backend in their code. GPU and system engineers can access the lower level C++ API. They can construct custom GpuModel compute graphs. They can write custom shaders. This allows for deep integration. Developers should test their models on the new engine. They should compare memory usage and latency. They should check if the performance gains match the benchmarks. If the app uses segmentation, they should look for the 40% latency reduction. If the app uses computational photography, they should look for the two-second speedup. They should also consider the supported APIs. If their app needs to run on Windows, Linux, and macOS, ML Drift with Dawn and Metal is a good fit. They should check if their hardware is supported. ML Drift supports GPUs from Arm, Intel, and Qualcomm. The Apache 2.0 license allows them to modify the engine if needed. They can contribute to the project or fork it for their own use. The migration is a one-time change. Once the code is updated, the app will benefit from the new engine.

Source: https://www.infoq.com/news/2026/10/google-ml-drift/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=AI%2C+ML+%26+Data+Engineering