LLM servers group requests to save money and speed up responses. This process is called batching. Imagine four people taking separate taxis instead of one bus. Four vehicles travel the same path but use more fuel. A single bus carries everyone and uses less fuel. AI servers work like the bus. They group multiple requests together for one processing step. This helps the server serve more users at once. It also makes better use of expensive computer chips. The speed of the user depends on how the server groups these requests. The cost to serve the user also depends on the grouping strategy. There are three main ways to do this grouping. These methods are static batching, dynamic batching, and continuous batching.
LLM Batching Explained: Static vs Dynamic vs Continuous Batching - The three main strategies for grouping requests
Batching means running many inputs through a model in a single step. Instead of handling one request at a time, the server handles many together. In a Large Language Model, each input is a list of words called tokens. The model processes a group of these lists together. Conceptually, the input changes from a single list to a larger group. This allows the computer chip to do math across many requests at once. It improves how busy the hardware is and how many tasks it finishes. However, different requests take different amounts of time to finish. The server must decide how to group them and when to free up resources. This decision determines how fast users get answers. It also determines how much it costs to run the model.
How Batching Works - Prefill and decode phases and why memory matters
LLMs have two main phases when they process a request. The first phase is called prefill. The model reads the entire prompt in one go. All the words in the prompt are processed together. This keeps the computer chip very busy during this time. The second phase is called decode. The model generates one word at a time. Each new word requires a full read of the model's brain. This means reading every weight from the chip's memory again. This phase is where wasting time happens. Some numbers show how slow this can be. A specific chip called an H100 has very fast memory. Reading a large file on this chip takes about 4 milliseconds. This limit caps how many words a single request can get in one second. It is around 240 words per second. Each word needs about 14 billion math operations. Doing this in 4 milliseconds uses only a small part of the chip's power. So, if only one request runs, the chip sits idle most of the time. If you run 32 requests together, the chip loads the weights once. You get 32 words instead of just one. This is how batching increases speed and lowers cost.
