The deployment of human-AI teams raises questions about whether to maintain the workflow or replace it with a human alone or the agent alone. The value of a human-AI workflow is dependent on it beating both alternatives. Replaying tasks under different alternatives consumes expert time or compute, and a fixed budget dictates which tasks receive human-only or agent-only replays. Existing evaluation methods do not directly address this design problem. TEAM-Design proposes a rule that assigns two replay probabilities to each task, increasing the probability for comparisons where the outcome is difficult to predict and decreasing it for expensive replays. The rule is proven to solve the budgeted design problem, and random replay selection still controls the risk of incorrectly declaring the workflow superior. The approach was tested across six clinical settings and a coding benchmark, along with synthetic and semi-synthetic designs. TEAM-Design performs best when one comparison is significantly harder to settle than the other.
Source: https://arxiv.org/abs/2609.05527