The research introduces a turn release scheduling method designed for agentic LLM workflows. These workflows involve sequences of model turns combined with tool interactions, and their completion time is influenced by when turns are released. The current eager release policy, where turns are submitted immediately upon readiness, can lead to accumulated unfinished work and increased tail latency under contention. The proposed method addresses this by jointly deciding which turn to release and managing the amount of unfinished work maintained. It utilizes a mean--Conditional Value-at-Risk (CVaR) objective to account for tail risk and incorporates online estimates of turn work for prioritization. The released work budget adapts to observed queue pressure. Evaluation using real traces from software engineering tasks across multiple LLMs and varying arrival rates demonstrated comparable performance to eager release under light load. Under contention, the method reduced the P95 of workflow flow time by up to 3.50 times.
Key details include the use of a CVaR objective, online turn work estimation, and adaptive work budget management. The method’s performance was assessed using real agent execution traces from software engineering tasks. The evaluation considered multiple LLMs and varied workflow arrival rates. The results highlight the importance of considering tail risk when scheduling agentic LLM workflows.
The research focuses on practical considerations for running agentic LLM workflows in production. Reducing tail latency is critical for improving the responsiveness and efficiency of these systems. The method provides a framework for managing contention and optimizing workflow completion times.
Source: https://arxiv.org/abs/2609.10964