DeepSeek releases DeepInfer open source inference stack

Architecture and scheduling details

DeepInfer separates the vision encoding, prefill, and decode stages. Each stage scales independently based on its own bottlenecks. The system recommends eight accelerators for prefill machines. Decode operations use 64 to 128 accelerator supernodes. Wide expert parallelism distributes MoE experts across these nodes. This design maintains large effective batches for better performance. The scheduling logic estimates compute cost rather than token count. Chunk sizes vary to fit a specific time budget. The system avoids enforcing a single fixed chunk size. Prefill routing does not require strict cache affinity. Requests can go to any available worker in the pool. Prefill data-parallel groups balance based on compute tokens. Shared storage supplies the reusable state for these groups. Decode routing balances both running request counts and KV-cache length. Request count approximates the scheduling pressure on the system. Accumulated KV length captures memory and state pressure accurately. The engineering commentary highlights these aggregate compute load signals. Expert-parallel load balancing replicates popular experts frequently. Replicas redistribute periodically to reduce inter-device waiting times. This trade-off consumes additional HBM memory for higher throughput. The useful replication factor depends on the specific workload. CPU scheduling overlaps accelerator execution times effectively. Asynchronous scheduling removes idle gaps between decode steps. This approach treats host-side orchestration as a paid resource. One observation notes the apparent absence of tensor parallelism. This configuration traces back to DeepSeek V3's H800-era constraints. Commentators suggest this is an interpretation rather than proof. Scale matters for wide expert-parallel prefill needs. Pipeline parallelism favors small-to-medium deployments with few workers. Layer heterogeneity can complicate the partitioning of pipeline stages. Practitioner judgment determines the optimal crossover point here.

Cache hierarchy and state management

The cache design connects throughput figures to revenue per accelerator. Persistent block KV lives in a separate 3FS storage cluster. This cluster holds roughly 100 PiB of SSD capacity. Reported retention is three days for agent traffic specifically. Chatbot traffic enjoys a seven-day retention window. Any prefill worker can retrieve a cached prefix from storage. This decouples cache locality from compute placement decisions. Storage and compute resources can scale independently now. This retention is useful for agents that resend history. Agents often repeat growing histories of instructions and tool outputs. The 155 trillion tokens daily must not be read as new. Cache hits are part of the service's token accounting. Newly computed prefill and autoregressive output impose different costs. The reported 96 percent input-cache hit rate is essential context. The implementation addresses two different state lifecycles clearly. Persistent block cache loads asynchronously into the system. Host-to-device transfers overlap layer execution during this process. Fine-grained KV block copies perform poorly through the CUDA engine. The described path uses GDR or RDMA-based transfers instead. This aims to saturate PCIe bandwidth without occupying SMs. Sliding-window attention state stays in host memory. This state remains useful mainly within active sessions. It has a shorter lifetime than reusable block prefixes. SWA Bounded Replay reconstructs missing short-lived state from 128 tokens. Replay restricts attention to that specific recent segment. The system reuses the persistent block cache for this step. This produces an approximation rather than rebuilding full state. DeepSeek reports this nearly halves total prefill computation. The approximation and its model-specific validity matter for portability. Long retention tails buy only approximately 0.5 to 1 percentage point. This illustrates a storage capacity versus recomputation trade-off. Decode-led cache management allows generated output to become reusable. Generated output becomes prefix state on subsequent turns. This timing invites comparison with OpenAI's 12-hour API caching. OpenAI announced a 30-minute default caching duration. These are not identical products or architectures. Both target persistent agents that reuse context across long gaps. Claims about indefinite logged-in caching are commentary, not terms.

Why it matters for cost and speed

DeepSeek's stack proves that open-source inference can scale massively. Teams can now study the code to improve their own systems. Lower compute costs mean cheaper access to AI capabilities. High margins show that efficiency drives profitability significantly. Speed improvements reduce latency for end-user applications. The 80 percent margin suggests huge potential for savings. Other companies might adopt similar caching strategies soon. The 96 percent cache hit rate is a key efficiency driver. Reducing token costs directly impacts the bottom line. Engineers can learn from the distributed inference techniques used. The separation of stages allows better resource allocation. Scheduling improvements reduce idle time on expensive GPUs. The architecture supports both NVIDIA and Huawei Ascend platforms. This flexibility helps teams choose the right hardware. The open source nature lowers the barrier to entry. Readers can run the code on their own clusters. Evaluation results will show how well it performs in real life. The transparency of the code builds trust in the system. Cost savings become achievable for smaller organizations too. Speed gains improve the user experience for every application. The combination of speed and cost is a rare win. This release sets a new standard for inference engineering. The community can now build on top of this foundation. The economic model shows the value of aggressive caching. Investors and managers can see the path to profitability. Developers get a clear view of the production stack. The open source movement gains a powerful new tool.

How OpenSmartRoute helps

A team routing requests through OpenSmartRoute gains immediate visibility into these costs. The router scores every candidate on quality, cost, speed, and safety. The team sets the weights for each specific request type. Hard rules pin requests to on-premises models for personal data. Region or cost cap rules ensure no single request crosses a limit. The system learns from outcomes to adjust traffic distribution. A model that answers well gets more traffic automatically. A model that fails gets less traffic the next time. A new model is just one catalogue entry in the app. It competes on the next request without changing the app. The router works with any OpenAI-compatible provider available. It supports open-weight models served locally on-premises. MCP tools and A2A agents integrate seamlessly into the flow. A savings ledger shows what each routed request actually cost. The ledger compares this to what the most expensive model would have cost. This transparency helps teams make data-driven routing decisions. The input guard spots prompt injection before a request leaves. This prevents security risks during the inference phase. The 'osr eval' tool measures routing accuracy on the team's own prompts. It can fail a build if the routing accuracy drops too low. The hosted platform keeps a models catalogue with prices. Public rankings are built from real traffic data automatically. This team can now benchmark DeepInfer against their current stack. They can see if the new architecture offers better margins. The router allows them to test the 96 percent cache hit strategy. They can route more traffic to the most efficient models. The savings ledger will show the dollar value of these improvements. The team can set up rules to enforce the new efficiency standards. OpenSmartRoute provides the infrastructure to exploit these new capabilities. The integration requires no complex setup or custom coding. The platform handles the complexity of multi-model orchestration. This team can adopt the new stack with minimal friction. The combination of OpenSmartRoute and DeepInfer creates a powerful system. The result is lower costs and higher reliability for all users.

What to do next

Engineers should clone the DeepInfer repository to inspect the code. They can run the basic examples on a local cluster. Comparing the throughput numbers to their own metrics is a good start. Check if the 96 percent cache hit rate is achievable in their environment. Review the scheduling logic to see how it handles heterogeneous requests. Test the SWA Bounded Replay mechanism for their specific use cases. Evaluate the cost implications of the 80 percent margin claim. Managers should review the revenue per accelerator-hour figures. Ask how these numbers translate to their own hardware budget. Consider if the 81 day payback period matches their financial goals. Plan a pilot project to test the new inference stack. Use the OpenSmartRoute platform to integrate the new models. Define the catalogue entries for the new DeepSeek-V4.1-Flash model. Set up the input guard to protect sensitive data from exposure. Configure the 'osr eval' tool to monitor routing performance daily. Track the savings ledger to measure the impact of the new stack. Compare the latency improvements against your application's requirements. Document the changes in your architecture for future reference. Share the findings with your team to drive broader adoption. Watch for updates to the repository as DeepSeek adds more features. Keep an eye on the community discussions for additional insights. The open source nature means you can contribute back to the project. Stay informed about the latest optimizations and performance tweaks. The goal is to build a more efficient and cost-effective AI infrastructure. This release is a significant step forward for the industry. The community will likely see many forks and variations of this stack. Prepare your team to handle the increased complexity of the new tools. The future of AI inference looks more open and accessible. Embrace the changes and leverage the new capabilities available. The path to lower costs and higher speed is now clearer.

Source: https://www.latent.space/p/ainews-deepseek-deepinfer-open-source