The research addresses the challenge of rapidly increasing climate literature and the difficulty in systematically identifying evidence of social tipping points. The paper details the creation of a modular framework designed to detect and organize these passages at the document level. The framework incorporates a DistilBERT boundary splitter for initial document segmentation, followed by an iteratively augmented RoBERTa classifier for detection. Subsequently, a Mistral 7B model is used to rewrite detected passages for improved clarity. A LLaMA 3.2 3B model then assesses the passages against five published social tipping point criteria. Finally, a Milvus vector store is employed for semantic retrieval.
The system was evaluated on a benchmark of 163 passages labeled by GPT-4.1 and a smaller set of 51 passages reviewed by experts. Performance metrics demonstrate the framework’s effectiveness. The splitter achieved a composite score of 6.137, surpassing three competing methods. The tuned RoBERTa model achieved 71.4 percent accuracy with a Cohen's kappa of 0.337 on the full benchmark, and 87.5 percent accuracy with a kappa of 0.742 on passages with labels.
The framework’s architecture allows for modular updates and experimentation with different models. The use of a vector store enables efficient semantic search and retrieval of relevant passages. The Streamlit interface and MinIO object storage provide a deployable solution. The system’s performance highlights the potential for automated analysis of large volumes of climate literature to accelerate research and decision-making.
The evaluation demonstrates the system’s ability to identify and structure evidence of social tipping points with a high degree of accuracy. This capability offers a valuable tool for researchers and practitioners seeking to understand and respond to complex social and environmental systems.
Source: https://arxiv.org/abs/2609.12254