StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Models
StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Models
Yongrui Chen, Yangyang Ma, Xiaoying Huang, Shenyu Zhang, Huajun Chen, Haofen Wang, Guilin Qi
Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence
Main Track. Pages 3792-3800.
https://doi.org/10.24963/ijcai.2026/422
Static benchmarks for LLMs are increasingly compromised by contamination and overfitting, especially on knowledge-intensive reasoning tasks. While recent dynamic benchmarks can alleviate staleness, they often increase difficulty at the expense of answerability and controllability. In this paper, we propose StressEval, a failure-driven data synthesis framework that turns observed model failures into dynamic, challenging, and controlable test instances. StressEval consists of three stages: (i) it constructs a semi-structured difficulty card that identifies the failed reasoning step and its root cause; (ii) it applies a dual-perspective instance-synthesis method that targets both knowledge gaps and reasoning breakdowns while preserving the underlying difficulty factors; and (iii) it applies a gating mechanism to retain only grounded, unambiguous instances. Seeding from multiple knowledge-intensive reasoning datasets, we employ StressEval to build Dynamic-OneEval, a focused suite of challenging dynamic benchmark. Across several state-of-the-art LLMs, Dynamic-OneEval yields substantially larger performance drops than the original benchmarks while retaining explicit difficulty factors, enabling more actionable iteration.
Keywords:
Knowledge Representation and Reasoning: Learning and reasoning
Natural Language Processing: Applications
Natural Language Processing: Question answering
