Scaffold presents a novel approach to building visual web agents. The framework induces parametric, executable skills from successful trajectories, creating a recursively composed skill hierarchy. This hierarchy allows for the invocation of lower-level skills by higher-level ones, enabling complex task execution. The system employs a minimum-description-length (MDL) criterion and behavioral equivalence checking to compact the skill library, reducing redundancy. Periodic distillation of skill-augmented trajectories back into model weights facilitates the internalization of abstractions.
Experiments conducted across WebArena, VisualWebArena, and a held-out Online-Mind2Web dataset demonstrate significant improvements. The Scaffold framework achieved an absolute success rate increase of $11.1$--$17.2$ points compared to the strongest skill-augmented baseline. These gains were observed across five self-improvement iterations, indicating sustained performance improvements without the degradation typically associated with skill libraries.
The core innovation lies in the recursive composition of skills. This hierarchical structure allows the agent to manage complexity and leverage previously learned knowledge effectively. The MDL compression further enhances efficiency by minimizing the storage requirements for redundant skills. This approach contrasts with previous skill-augmented frameworks that treat the skill library as a simple cache.
The code and documentation are available on Github. Source: https://arxiv.org/abs/2609.05511v1