Exploring the System 1 Thinking Capability of Large Reasoning Models

1 Introduction

Recent advances in Large Reasoning Models (LRMs), notably OpenAI's o1/o3 [OpenAI, 2024] and the DeepSeek-R1 [Guo et al., 2025] series, have propelled the development of Large Language Models (LLMs). Unlike traditional LLMs that exhibit intuitive, heuristic system 1 thinking, LRMs demonstrate deliberate and analytical system 2 reasoning [Qu et al., 2025; Li et al., 2025c] by explicitly generating external chain-ofthought [Wei et al., 2022] before producing final answers.

While most prior research focuses on enhancing LRMs' reasoning capabilities to achieve superior performance on complex tasks requiring system 2 [Yang et al., 2025; Li et al., 2025a; Yeo et al., 2025], exploration of their system 1 thinking abilities remains limited. Specifically, system 1 thinking refers to the human-like intuitive ability [Kahneman, 2011] to answer questions rapidly with minimal token usage. This capability reflects a model's perception of task difficulty as well as its proficiency in efficiently responding to questions it recognizes as simple [Booch et al., 2021]. In real-world scenarios, where the vast majority of user queries are relatively straightforward for models, ensuring accurate and efficient system 1 thinking is particularly crucial [Yang et al., 2024].

In this work, we aim to explore the system 1 thinking capabilities of LRMs, yet a suitable benchmark remains absent. Some studies attempt to evaluate LRMs using questions that humans consider simple (e.g., GSM8K-zero and RoR [Chiang and Lee, 2024; Yan et al., 2025]), yet these questions remain challenging for models. Other works utilize single-domain benchmarks, such as basic mathematics (GSM8K and ASDIV [Cobbe et al., 2021; Miao et al., 2020]), but lack domain diversity. Other research on LRM efficiency primarily targets challenging problems, such as AIME. As shown in Table 1, the top six benchmarks which are most commonly used in these work remain difficult for small LLMs, making them unsuitable for exploring system 1 thinking capabilities.

To complement the exploration foundation, we introduce the System 1 Thinking Capability Benchmark (S1-Bench), which measures the performance of LRMs across various simple tasks that are commonly encountered in real-world applications. S1-Bench has the following three characteristics: (1) Simple. The questions can be easily answered by LLMs. As shown in Table 1, LLMs with 7-9B parameters can robustly provide correct answers across different temperatures. (2) Diverse. It encompasses four major categories and 28 subcategories in two languages. (3) Natural. The questions are clear, without any misleading elements or ambiguities, ensuring they can be answered intuitively.

We conduct extensive exploration on S1-Bench across

28 LRMs, yielding the following key findings: (1) Underaccuracy. Despite employing deep reasoning, several LRMs exhibit under-accuracy on simple questions, particularly in knowledge-based and instruction-following tasks, suggesting that long-chain reasoning may introduce catastrophic forgetting. (2) Inefficiency: Current LRMs lack system 1 capabilities across all categories of questions, with average output lengths 13.8× longer than small LLMs on S1-Bench. (3) Limitations of efficient reasoning methods: Existing efficient reasoning methods either suffer from limited generalization, failing to effectively compress length on S1-Bench, or fail to balance performance and efficiency, leading to significant drops in accuracy. (4) Early awareness of problem difficulty: We observe that LRMs often exhibit explicit difficulty-related comments prior to formal reasoning, indicating that models possess an early awareness of problem difficulty. Moreover, even when identifying a question as simple, LRMs still generate redundant responses with higher average token entropy, indicating stronger exploratory. These reveal a gap between models' difficulty awareness and reasoning behavior. Furthermore, our representation-level analysis shows that question difficulty is implicitly encoded in model hidden states, as a simple single-layer MLP can capture difficulty-related signals. Our contributions can be summarized as follows:

• We propose S1-Bench, a benchmark dedicated to evaluating LRMs' system 1 thinking capabilities, and a semiautomated construction workflow for system 1 questions.
• We reveal LRMs' under-accuracy and inefficiency in system 1 thinking, and highlight the inadequacy of current efficient reasoning methods, informing future research.
• We explore LRMs' early difficulty awareness, revealing correlations between explicit responses and model confidence, and disentanglement of difficulty in hidden states.

2 S1-Bench

We introduce S1-Bench, a bilingual, multi-domain benchmark designed to evaluate system 1 thinking capability of LRM on extremely simple questions. These questions are easily solvable by traditional LLMs. S1-Bench, which covers both English and Chinese, is organized into four major categories: reasoning (RSN), knowledge (KNO), instruction following (IF) and analysis (ANA), representing major dimensions commonly employed in LLM capability evaluation [Zheng et al., 2023;

Chang et al., 2024]. This section begins with how simplicity is ensured, then the detailed construction workflow for S1Bench, and concludes with an overview of the dataset statistics. Figure 1 shows the construction workflow and an illustrative example per category.

2.1 How to Ensure Simplicity?

We ensure questions are simple and suitable for system 1 thinking through the following two aspects.

A Priori Simplicity Constraints

We begin by generating question-answer pairs through collaboration between humans and LLMs. Each pair is required to satisfy both the general and the category-specific simplicity criteria. The general simplicity criteria requires that: (1) Questions must be naturally and clearly expressed, unambiguous, and free of intentional traps. (2) Answers must be unique or easily falsifiable (e.g., providing a three-letter English word).

The category-specific simplicity criteria are as follows. RSN: Limited to problems solvable with minimal reasoning or intuition. KNO: Restricted to common knowledge with unique, verifiable answers from sources likeWikipedia. IF: Involve straightforward instructions without strict formatting requirements. ANA: Limited to questions whose answers can be directly inferred from the prompt, such as binary classification. These constraints ensure all questions remain straightforward for human respondents.

A Posteriori Simplicity Verification

Due to the biases existing between language models and humans [Gallegos et al., 2024], questions that are simple for humans may be difficult for LLMs. Therefore, we introduce additional posteriori verification to ensure that questions are simple enough to be correctly and robustly answered by smaller LLMs from different families.

2.2 ConstructionWorkflow

Subcategory Preparation. To ensure diversity, we refer to the subcategories included in existing benchmarks (e.g., MMLU, IFEval, and GSM8K) and evaluation surveys [Chang et al., 2024] to select, merge, or design subcategories for S1bench, ensuring that each meets the simplicity requirements.

Implementation of A Priori Simplicity. First, we use two data generators 1 to create 100 initial bilingual questionanswer pairs for each candidate subcategory. The data generation prompt explicitly incorporates the subcategory definitions, along with both the general and category-specific simplicity criteria, while also aiming to ensure diversity in the generated questions. Second, these question-answer pairs are then independently evaluated by three annotators and two quality discriminators 2 according to the general and category-specific simplicity criteria, resulting in five evaluation outcomes per pair. The three annotators are experienced graduate students familiar with LLMs and well-acquainted with the goals of S1-Bench. Finally, based on these evaluation outcomes, three annotators discuss and collectively decide whether to retain, modify, or discard each question.

Implementation of A Posteriori Simplicity. First, each question obtained from the previous stage is input into the small LLM validators 3 with 7˜9 B parameters. For each question, we sample 10 answers at three different temperature settings (0, 0.2, and 0.4), resulting in a total of 30 responses per question. These responses are then individually evaluated for correctness using GPT-4o. Second, if all 30 sampled responses are correct, the question is accepted into S1-Bench. Otherwise, the question is returned to the generators, where a difficulty-reduction prompt is applied to simplify it. The simplified question then undergoes the same subsequent process. Finally, questions that fail to meet the full-accuracy criterion (i.e., 30 out of 30 correct) after three rounds of difficulty reduction are excluded from the workflow.

The final S1-Bench comprises questions validated by both human a priori constraints and LLM a posteriori verification.

2.3 Benchmark Statistics

S1-Bench comprises 422 question-answer pairs across four major categories and 28 subcategories, balanced with 220 English and 202 Chinese questions, with questions averaging 14.46 tokens. To ensure that the a posteriori verification process does not introduce simplicity only tailored to the small LLM validator, we evaluate S1-Bench on five additional LLMs and on Qwen3 Family with reasoning modes disabled. As shown in Table 2, even the 1.7B model achieves over 98% accuracy at temperature 0.

1 We select Claude-3.7-Sonnet and Qwen2.5-72B-Instruct.

2 We select GPT-4o and DeepSeek-V3-241226.

3 Qwen2.5-7B, Llama3.1-8B, Mistral-8B, Gemma2-9B.

3 Experiment Setting

3.1 Baseline Models and Configurations

We evaluated 28 different LRMs, which are explicitly trained to first respond with a thinking process, and then generate a final answer. The evaluated LRMs include both open-source models-such as DeepSeek [Guo et al., 2025], Qwen [Yang et al., 2025], Nemotron [Singhal et al., 2025], Light-R1 [Wen et al., 2025], and s1.1 [Muennighoff et al., 2025]-and closedsource models including claude-sonnet-4-20250514, claude-37-sonnet-20250219, gemini-2.5-flash-preview-09-25, o4-mini, and Hunyuan-T1 4 , covering parameter scales from 1.5B to 671B. For each model, we adopt a temperature of t = 0.6, topp = 0.95, and a sampling size of k = 5, with the maximum generation length set to 10,000 tokens.

3.2 EvaluationMetrics

FormatMetrics. We measure format accuracy (f-acc) as the proportion of responses that follow the required output format, averaged over five runs. A response is considered valid if an end-of-thinking marker (e.g., </think>) correctly separates the reasoning process from a non-empty final answer.

4 https://llm.hunyuan.tencent.com

EfficiencyMetrics. We report the average response length in tokens (Tokens), excluding samples with endless thinking, with all token counts computed using the Qwen2.5 tokenizer to ensure fair comparison.

AccuracyMetrics. We calculate accuracy metrics only for responses with correct formatting. Following the LLM-asjudge setting [Zheng et al., 2023], we use GPT-4o as the evaluator to assess the final answer part. We use two metrics: pass@1 and acc@k. pass@1 follows the definition in DeepSeek-R1 [Guo et al., 2025], computing the average accuracy over k samples, while acc@k is the percentage of questions where all k answers are correct. Both metrics use k=5. Notably, f-acc represents the upper bound for pass@1 and acc@k under the formatting requirement.

4 Main Experiment

In this section, we conduct a comprehensive evaluation of LRMs on S1-Bench from three perspectives: accuracy (Table 3), efficiency (Table 5), and the impact of RL-based efficient reasoning algorithms (Table 6).

4.1 Accuracy Analysis

We analyze the accuracy-related results, and as shown in Table 3, we obtain the following observations:

Although the vast majority of larger-scale or closedsource models can maintain the same level of accuracy as small LLM-based validators (often exceeding 99% accuracy), some smaller LRMs suffer from severe performance degradation. For example, DS-R1-1.5B and Nemotron-8B achieve acc@k values only slightly above 50%. This behavior directly contradicts the original goal of LRMs: scaling thinking length can actually undermine accuracy on System 1 problems. Moreover, many LRMs exhibit limited robustness, with pronounced gaps between pass@1 and acc@k consistently observed across models, including DS-R1-1.5B (↓26.97%), L-R1-32B (↓15.12%), DS-R1-7B (↓9.43%), LR1-7B-DS (↓10.99%), Nemotron-8B (↓20.81%), and s1.1-7B (↓24.60%). Finally, certain models suffer from output format misalignment, as evidenced by format correctness below 90% for Nemotron-8B and s1.1-7B.

Accuracy degradation is particularly pronounced on instruction-following and knowledge-based questions. We conduct an question type analysis on models whose pass@1 falls below 90%. As shown in Table 4, a clear performance gap emerges across categories: instruction-following and knowledge questions reach average accuracies of only 76.7% and 78.6%, respectively, versus 88.1% on reasoning problems. This pattern suggests that long-chain reasoning does not consistently improve model behavior and may instead weaken instruction-following ability and factual reliability.

We further analyze the error cases and types in the responses. Specifically, we use DeepSeek-v3 to categorize LRM responses into four cases and compute their proportions. (1) Final answer correct; thinking process entirely accurate. (2) Final answer correct; thinking process contains intermediate errors. (3) Final answer incorrect; correct answer mentioned in thinking process. (4) Final answer incorrect; correct answer never mentioned in thinking process. Results are shown in Figure 2. Key findings include: (1) Lower-accuracy LRMs tend to produce less reliable reasoning chains; even when they arrive at the correct final answer, their intermediate steps often contain errors (light green). LRMs with high accuracy (e.g., DS-R1) show almost no flawed reasoning steps, whereas those with lower accuracy (e.g., DS-R1-1.5B) often generate incorrect intermediate conclusions, further indicating that they lack robust reasoning ability. (2) Although LRMs sometimes mention the correct answer during reasoning, they may deviate and ultimately produce incorrect final answers (light red). For example, in one case, the LRM initially arrived at the correct answer but undermined it through excessive verification. In another case, the LRM directly denied the correct answer that appeared during reasoning.

4.2 Efficiency Analysis

As shown in Table 3, we present the average response tokens for each problem category in S1-Bench sampled five times (2.11k sample points per model). We observe that: (1) all LRMs exhibit overthinking on System 1 problems. While closed-source LRMs show higher efficiency, they still consume over three times more tokens. Open-source LRMs are considerably less efficient, with s1.1 and Light-R1 family models exhibiting severe overthinking to achieve better performance on complex problems. (2) LRMs within the same family show similar efficiency without demonstrating clear scaling phenomena. Across DS, Qwen3, and Nemotron, response lengths cluster by model family rather than by size, showing no clear correlation between model scale and thinking length. This suggests that training methods and data composition may be more critical factors in determining System 1 thinking effective.

Analysis Across Question Types

We further present the average response tokens by fine-grained categories and languages, as shown in Table 5. The main findings are as follows: (1) LRMs exhibit a substantial increase in response length across all four major categories, 28 subcategories, and two languages. As shown in Table 5, for each of the four major categories, the average response length of LRMs exceeds that of LLMs by more than a factor of ten. Response lengths also increase significantly across all subcategories. This suggests that while LRMs are primarily trained on reasoning data to produce long CoT style responses, this stylistic pattern generalizes well across a wide range of question types. (2) LRMs exhibit the most significant increase in tokens for instruction following questions and tend to over-explore when the solution space is vast. As shown in Table 5, although small LLMs provide the most concise responses to instruction following questions, LRMs generate dramatically longer outputs-82.1 × longer in English and 49.9 × longer in Chinese than small LLMs. To investigate the cause, we further analyze the subcategories of instruction following questions. Average tokens is notably longer in the subcategories of length constraints, character constraints, and sentence constraints. These three question types share a similar characteristic: their correctness is easy to verify, but the solution space is vast. We find that, although the model quickly identifies a correct answer, it becomes trapped in the search space, continually exploring alternatives and failing to stop in time.

Thinking Solution Analysis

To better understand the causes of inefficiency in LRMs on S1-Bench, we analyze the solution rounds of their thinking processes. We first use DeepSeek-v3 to segment each thinking process into several solutions, each defined as a point at which LRMs explicitly arrives at a conclusion that matches the correct answer. We then compute the average token counts in the first solution. Our analysis reveals the following: (1) The token consumed in the first solution of LRMs significantly exceeds that of validator LLMs. As shown in Figure 3 (a), this suggests that LRMs may involve unnecessary reasoning steps in first solution, which could be one of the reasons for their inefficiency. (2) The primary reason for efficiency gaps between LRMs lies in the number of redundant solution rounds they generate, rather than the token cost in the initial round. As shown in Figure 3 (a), although total thinking token counts vary widely across LRMs, their token counts in the initial round are similar and only account for a small fraction of the total. Figure 3 (b) further shows the distribution of solution rounds on S1-Bench, revealing that LRMs with longer thinking processes tend to generate more solution round, and this redundancy greatly increases computational cost. Furthermore, further experiments reveal that the redundancy in the reasoning process gradually increases over time. Furthermore, the similarity across solution rounds progressively increases as reasoning proceeds, suggesting that later rounds contribute increasingly redundant information.

4.3 Efficient Reasoning Algorithms Analysis

Recently, several methods have been proposed to mitigate overthinking in LRMs based on reinforcement learning (RL). We evaluate representative approaches on S1-Bench, including (1) RL with length-based penalties (e.g., Shorterbetter [Yi et al., 2025], Laser-D [Liu et al., 2025], and TLMRE [Arora and Zanette, 2026]), and (2) adaptive mode selection (e.g., Adaptthink [Zhang et al., 2025a] and Autothink [Tu et al., 2025]). Table 6 reports results for models fine-tuned from DSR1-1.5B/7B, leading to the following observations. (1) Token reduction on challenging benchmarks does not necessarily translate to similar gains on S1-Bench. For instance, LaserD-1.5B and AdaptThink-1.5B reduce reasoning tokens by nearly 50% on AIME2024, but achieve only about 25% reduction on S1-Bench, indicating limited generalization of these methods to out-of-distribution problems. (2) Some methods, such as ShorterBetter and AutoThink, achieve substantial compression on S1-Bench at the cost of severe accuracy degradation. Specifically, ShorterBetter-1.5B attains only

18.9% pass@1 on AIME24, while AutoThink-1.5B achieves merely 60.1% accuracy on S1-Bench, suggesting difficulty in balancing reasoning efficiency with answer correctness.

5 Early Difficulty Awareness in LRMs

We observe that on S1-Bench, LRMs may form an early intuitive judgment of question difficulty before initiating formal reasoning. In some cases, models begin their responses with statements such as "This looks like a simple problem," prior to any substantive reasoning. We refer to this phenomenon as early awareness of question difficulty. To better understand this behavior, we conduct the following analyses.

Exploring Frequency and Stylistic Patterns of Difficulty Awareness. We employ GPT-4o to classify the initial portion of model responses-defined as the content before the first "\n\n"-based on whether and how the model comments on question difficulty. Each response is categorized into one of four types: easy, neutral, difficult, or no comment. Figure 4 presents the distribution and corresponding probabilities of these categories across four types of questions. From these results, we make several observations. First, all LRMs exhibit early difficulty awareness to varying degrees, with the phenomenon being particularly prominent in the Qwen, DeepSeek, and Light-R1 model families. Second, LRMs display clear stylistic differences in expressing such awareness. For instance, the Qwen family tends to characterize questions in S1Bench as easy, whereas DeepSeek-distilled models produce more diverse difficulty-related comments. Third, this early difficulty awareness is most evident in reasoning-oriented questions, while it appears less frequently in analytical questions.

Exploring the Relationship Between Early Difficulty Awareness, Reasoning Length, and Entropy. To investigate whether the early sense of a question as "easy" leads to a corresponding reduction in response length, we compare the average response tokens for questions in the easy category versus all samples. The results are shown in Figure 5. Except for L-R1-32B, the remaining LRMs do not exhibit a noticeable decrease in response length when questions are perceived as easy. On the contrary, 18 out of 23 LRMs generate longer responses under this condition. We further analyze the average token entropy of responses associated with easy judgments, with the results presented in Figure 6. Interestingly, these responses exhibit higher average entropy, indicating stronger exploratory behavior during generation. One possible explanation is that models may exhibit self-doubt toward their own early difficulty judgments, leading them to engage in increased exploration and produce longer responses, despite initially perceiving the question as simple. This suggests a discrepancy between the LRM's initial sense of difficulty and its generative behavior, the causes and improvements of which warrant further investigation.

Exploring Activation Pattern Differences Across Question Difficulty. In contrast to the preceding analyses based on explicit difficulty-related cues in model outputs, we further examine whether question difficulty is reflected at the representation level through systematic differences in model activation patterns. Specifically, using DS-R1-1.5B and DS-R1-7B as representative examples, we analyze whether the final-layer representation of the last token in the encoded question varies systematically with question difficulty. To this end, we conduct a probing-based analysis by training a single-layer MLP as a weak linear classifier with limited training data, which serves solely to assess whether hidden representations contain information predictive of question difficulty. The training set is constructed by selecting one Chinese and one English example from each subcategory of S1-Bench as easy questions, and sampling an equal number of bilingual questions from AIME25 as hard questions, where Chinese examples are obtained via translation from English. We then evaluate the trained probe on the MATH500 dataset, which naturally contains five difficulty levels. For each level, we compute the proportion of questions classified as difficult. As shown in Table 7, this proportion increases consistently with question difficulty, indicating that model representations implicitly encode structured information aligned with problem difficulty.

6 RelatedWork

6.1 Large ReasoningModels

Large Reasoning Models (LRMs), characterized by explicitly generating external thinking processes before final answers [Kumar et al., 2025b; Li et al., 2025b], achieve a paradigm shift from intuitive system 1 thinking to deliberative system 2 reasoning compared to traditional LLMs [Li et al., 2025c; Qu et al., 2025], thus achieving superior performance on complex tasks. The development of recent LRMs has largely followed two main approaches: largescale reinforcement learning (RL) and model distillation. Models trained via RL [Ai et al., 2026; Team et al., 2025; Nie et al., 2026] leverage reward-based optimization to gradually incentivize deliberative reasoning. In contrast, distillation-based LRMs [Min et al., 2024; Ye et al., 2025; Muennighoff et al., 2025; Zhang et al., 2025b] acquire such abilities by transferring structured reasoning patterns from advanced teacher models.

6.2 Limitations of LRMs

While LRMs have shown significant performance gains through deliberate reasoning, rigid adherence to this overly cautious thinking can introduce new limitations. On one hand, intermediate reasoning steps can cause excessive token generation and unnecessary solving attempts [Chen et al., 2024; Kumar et al., 2025a; Gong et al., 2026], even leading to redundancy in the hidden layers [Chen et al., 2025]. On the other hand, LRMs' performance can drop in specific contexts like safety scenarios [Jiang et al., 2025], agents [Zhang et al., 2026] and role-playing [Feng et al., 2025]. However, prior studies mainly evaluated LRMs on complex tasks that more suited for deliberative system 2 thinking. Our work examines how deliberative reasoning impacts extremely simple problems better matched to intuition-driven system 1 processing.

7 Conclusion

This paper extensively explores the system 1 thinking capabilities of LRMs. We construct S1-Bench, a simple, domaindiverse, and natural benchmark for evaluating system 1 thinking abilities. Our exploration reveals two fundamental issues of LRMs on system 1 problems, namely under-accuracy and over-reasoning, and exposes limitations of existing efficient training methods. We also demonstrate models' inherent perception of system 1 problems from both linguistic and representational perspectives, providing insights for future research.

Acknowledgments

We would like to thank the anonymous reviewers, the area chairs and program chairs for their valuable comments and efforts. This work is supported by the National Natural Science Foundation of China (Grant No. 62572465).

Contribution Statement

Equal contribution: Wenyuan Zhang and Shuaiyi Nie. Corresponding author: Tingwen Liu.