Toward Reliable Scientific Hypothesis Generation: Evaluating Truthfulness and Hallucination in Large Language Models

1 Introduction

Large language models (LLMs) have transformed the landscape of artificial intelligence, demonstrating remarkable capabilities across diverse applications, from natural language understanding to creative content generation [Karanikolas et al., 2023; Franceschelli and Musolesi, 2024; Raiaan et al., 2024]. These models, trained on extensive corpora of text, demonstrate an ability to analyze, summarize, and generate human-like text, enabling advancements across diverse domains. Recently, there has been a growing interest in leveraging LLMs for scientific discovery [Zhong et al., 2023; Yang et al., 2023; Kumar et al., 2023; Liu et al., 2024;

Baek et al., 2024; Si et al., 2024]. Their capacity to process and synthesize vast amounts of scientific literature positions them as valuable tools in aiding researchers, particularly for tasks such as literature reviews, summarization, and even generating new hypotheses [Qi et al., 2023; Zhou et al., 2024; M. Bran et al., 2024; Wright et al., 2022; Zeng et al., 2023; D'Arcy et al., 2024; Ifargan et al., 2025; Yang et al., 2025].

One particularly promising application of LLMs is their use in scientific hypothesis generation, where they can assist in identifying promising research directions. By analyzing extensive scientific literature, LLMs can uncover gaps in existing knowledge and propose novel hypotheses that may not be immediately apparent to human researchers. For instance, LLMs have been successfully applied to propose novel drug combinations for breast cancer treatment, some of which were later validated in laboratory experiments, showcasing their potential to accelerate biomedical discoveries [AbdelRehim et al., 2024].

Despite these advancements, there are substantial challenges that limit the practical utility of LLMs in scientific hypothesis generation. A critical concern is the inability to evaluate the truthfulness of generated hypotheses. While LLMs can generate hypotheses that seem plausible, it remains uncertain whether these hypotheses are valid and grounded in existing knowledge or merely hallucinated and scientifically invalid. This issue is further exacerbated by the welldocumented "hallucination" problem, where LLMs confidently produce information that is factually inaccurate or unsupported, posing challenges to their reliability in scientific contexts. While current research has largely focused on improving the novelty and diversity of LLM-generated hypotheses, their truthfulness and grounding in established knowledge remain underexplored [Baek et al., 2024; Hu et al., 2024; Si et al., 2024].

To address these challenges, we introduce TruthHypo 1 , a comprehensive benchmark for evaluating the ability of LLMs to generate truthful scientific hypotheses, and KnowHD, a knowledge-based hallucination detection framework designed to assess the groundedness of these hypotheses. TruthHypo, built on a biomedical knowledge graph along with a domain-specific corpus, provides a controlled environment to evaluate how well LLM-generated hypotheses align with established scientific knowledge. KnowHD focuses on analyzing the reasoning processes of LLMs to identify hypotheses that are likely hallucinated or untruthful. Our findings reveal that LLMs face significant challenges in generating truthful hypotheses. By analyzing hallucinations in the reasoning processes behind generated hypotheses, we demonstrate that groundedness scores from KnowHD serve as an effective signal for identifying truthful hypotheses from the diverse outputs of LLMs. Human evaluations on open-ended hypothesis generation tasks further confirm the utility of KnowHD in identifying scientifically valid hypotheses.

1 An extended version of this paper with Appendix is available at https://arxiv.org/abs/2505.14599.

Our main contributions are summarized as follows:

• We introduce TruthHypo, a comprehensive benchmark designed to evaluate the ability of LLMs to generate truthful scientific hypotheses.
• We propose KnowHD, a knowledge-based hallucination detection framework that assesses the groundedness of LLM-generated hypotheses and identifies hallucinated claims by analyzing the rationale behind the hypothesis generation.
• We provide an extensive analysis of existing LLMs on TruthHypo, highlighting their limitations and challenges in generating truthful hypotheses.
• Our evaluation further reveals the connection between hallucination and truthfulness of generated hypotheses, showing the effectiveness of using KnowHD to select truthful and grounded hypotheses.

2 Truthful Hypothesis Generation Benchmark

To systematically evaluate the ability of large language models (LLMs) to generate truthful scientific hypotheses, we introduce TruthHypo, a benchmark tailored for biomedical hypothesis generation. TruthHypo is designed to simulate realworld conditions by employing rigorous dataset construction, task formulation, and truthfulness evaluation metrics. An overview of the dataset construction, task formulation, and evaluation framework is depicted in Figure 1.

2.1 Dataset Construction

The dataset for TruthHypo is derived from PubTator 3.0 [Wei et al., 2024], a comprehensive biomedical knowledge graph that includes annotated relations (also called edges) extracted from scientific articles. To simulate the temporal progression of scientific discovery, we partitioned the graph into "seen" and "unseen" subsets based on the publication years of the corresponding articles. Relations in the "seen" subset were extracted from papers published before 2023, identified by PMIDs ≤ 366000002 . The "unseen" subset, designed to represent new discoveries, comprises relations extracted from papers published after 2024, identified by PMIDs ≥ 38200000.

To ensure no overlap between the two subsets, we removed the edges in the unseen subset that shared head and tail entities with those in the seen subset. In addition, to maintain quality and validity, only relations discovered by multiple articles in the test data were retained. This filtering process guarantees that the unseen subset exclusively contains knowledge unavailable before 2024, simulating the conditions of future scientific research.

In building the dataset, we focused on three key relation types: "Chemical & Gene", "Disease & Gene", and "Gene & Gene". These relation types were chosen for their complementary nature, detailed annotations, and potential for objective evaluation. To construct comprehensive classification tasks for evaluating different LLMs, we augment the dataset with negative test cases to assess whether LLMs tend to make false-positive predictions on entity pairs that lack a direct relationship in the existing knowledge base. The number of negative samples (labeled as "no relation") for each relation type is controlled to align with the average number of instances across other labels of the same relation type. The final dataset has 1209 instances for the "Chemical & Gene" task, 268 instances for the "Disease & Gene" task, and 547 instances for the "Gene & Gene" task. A summary of the dataset statistics is presented in Table 1.

2 PMID is the unique identifier of the paper where the edge was extracted.

2.2 Task Formulation

The TruthHypo benchmark includes three tasks, corresponding to the selected relation types: "Chemical & Gene", "Disease &Gene", and "Gene &Gene". For each task, the input is a hypothesis generation query with two entities, and the LLM is required to hypothesize the potential relationship between them based on available knowledge and reasoning.

To comprehensively assess LLM performance, we evaluate their ability to generate hypotheses under various knowledge augmentation settings. In the first setting, LLMs rely solely on their parametric knowledge - information encoded in their parameters during pretraining on large corpora. This evaluates the model's intrinsic understanding and reasoning capabilities.

To enhance hypothesis generation, we introduce a second setting in which LLMs are augmented with structured knowledge from the "seen" knowledge graph. In this approach, key entities from the input are mapped to nodes in the graph, and multi-hop link chains connecting these nodes are explored. These chains, representing relevant relationships, are transformed into textual descriptions and provided as context for the model to use during hypothesis generation.

Another setting leverages information from biomedical literature using a retrieval-augmented generation (RAG) pipeline. Relevant documents are retrieved from the PubMed corpus 3 using BM25 [Robertson et al., 2009]. To maintain consistency with the knowledge graph's temporal split, only articles with PMIDs ≤ 36600000 are included in the retrieval. This simulates the process of generating hypotheses based on literature available at a given point in time.

Finally, we consider a combined setting, where both structured knowledge from the graph and unstructured information from retrieved literature are used to support hypothesis generation. This comprehensive approach provides a more holistic context, enabling the model to reason across both sources. The LLM prompt templates we used to combine the external information with the original user instructions can be found in the Appendix.

3 https://pubmed.ncbi.nlm.nih.gov/

2.3 EvaluationMetrics

To evaluate the quality of generated scientific hypotheses, we employ a set of complementary metrics tailored to different aspects of hypothesis generation. These metrics assess the performance of LLMs in identifying valid connections between entities (link-level evaluation) and predicting specific relations (relation-level evaluation).

For link-level evaluation, we focus on precision, recall, and F1 score. Precision measures the proportion of correctly identified connections among all hypothesized connections, emphasizing the reduction of false positives. Recall evaluates the model's ability to comprehensively identify all valid connections, capturing its sensitivity to true positives. The F1 score, as the harmonic mean of precision and recall, provides a balanced measure of performance, combining both the accuracy of predictions and the coverage of valid connections. These link-level metrics are critical for assessing the LLM's ability to hypothesize plausible relationships between entities, regardless of the specific relation type.

For relation-level evaluation, we employ accuracy to measure how often the generated hypotheses match the correct relation labels in the ground truth. Accuracy captures the overall correctness of hypotheses by considering both the existence of a connection and the predicted relation type. While precision, recall, and F1 focus on identifying potential connections, accuracy provides a finer-grained assessment of the model's capability to generate accurate relation labels.

By combining link-level and relation-level evaluations, the TruthHypo benchmark comprehensively measures the truthfulness of LLM-generated hypotheses, assessing the ability of LLMs to produce scientifically valid outputs.

3 Knowledge-based Hallucination Detection

As discussed earlier, a critical concern regarding the truthfulness of LLM-generated hypotheses is the occurrence of hallucinations, where models generate plausible-sounding but unsupported claims. To address this, we introduce KnowHD, a knowledge-based hallucination detection framework that evaluates the groundedness of LLM-generated hypotheses by analyzing the rationale behind their generation. KnowHD operates using scientific literature, knowledge graphs, or a combination of both as the knowledge base. An overview of the framework is presented in Figure 2.

To evaluate groundedness, each hypothesis and its reasoning chain are first decomposed into a set of atomic claims. This step is critical because hypotheses often consist of compound reasoning steps, some of which may be supported by existing knowledge while others may not. Parsing these into atomic claims allows a more granular evaluation of groundedness and isolates unsupported components. This step is implemented by prompting LLMs with the template shown in the Appendix.

When using scientific literature as the knowledge base, relevant documents for each atomic claim are retrieved from the PubMed corpus, limited to articles published before 2023 (PMID ≤ 36600000). BM25 is employed to rank documents based on their relevance to the claim. To ensure computational efficiency and focus on the most relevant information, only the top-k documents are retained. The context retrieved from the literature corpus D for a claim p is defined as:

where di represents a document in the corpus, BM25(p; di) is the relevance score assigned to the document for the claim p. τ is a threshold ensuring relevance, and rank(di) denotes the rank of di in the BM25-retrieved list.

When using a knowledge graph G as the knowledge base, the context for a claim is derived from the graph structure. For a claim p, relevant knowledge is extracted as:

where (eh, r, et) represents an edge in the knowledge graph with head entity eh, tail entity et, and relation r. The set V(p) contains all entities mentioned in the claim p.

The groundedness of a claim is determined based on whether the given context information (contextD, contextG, or contextD ∪ contextG) can fully support the claim, which is implemented by prompting LLMs to provide a judgment using the template in the Appendix. If the concatenated context collectively entails the claim, it is considered grounded. The overall groundedness of a hypothesis h is computed as:

where C(h) represents the set of atomic claims for hypothesis h, and 1[x |= y] returns 1 if x entails y and 0 otherwise. The context(p) can be contextD(p), contextG(p), or contextD(p)∪ contextG (p).

By offering both literature-based and graph-based contexts, KnowHD provides a robust framework for hallucination detection, offering flexibility to adapt to the available knowledge sources. This systematic evaluation of atomic claims enables a detailed assessment of the groundedness of hypotheses, identifying unsupported components and improving the reliability of LLM-generated outputs.

4 Benchmark Analysis on TruthHypo

4.1 Experiment Settings

To assess the ability of existing LLMs to generate truthful scientific hypotheses, we selected a diverse range of models varying in type and size. The Llama-3 family [Dubey et al., 2024] represents open-source LLMs, while the GPT-4 family [Achiam et al., 2023] exemplifies proprietary models. From each family, we evaluated two LLMs of different sizes (Llama-3.1-8B & Llama-3.1-70B, GPT-4o-mini & GPT-4o) to investigate size-related differences in performance. All LLMs were trained on the knowledge available before 2024, preventing recall of the exact knowledge for hypothesis generation. More implementation details are in the Appendix.

The TruthHypo benchmark evaluates LLMs across four distinct settings: (1) parametric knowledge only, (2) parametric knowledge with knowledge graphs (KG), (3) parametric knowledge with literature (Lit.), and (4) parametric knowledge with both KG and literature. These settings allow us to explore the impact of external knowledge sources on hypothesis generation. The F1 and accuracy scores of different models are reported in this section. More detailed results on the precision and recall can be found in the Appendix.

4.2 Comparison of LLMs in Truthful Hypothesis Generation

Table 2 presents the evaluation results for different LLMs and knowledge settings on TruthHypo. Across all tasks, the results indicate that most LLMs struggle to generate truthful scientific hypotheses, with only GPT-4o achieving mean accuracies exceeding 60%. Additionally, we can observe that link-level F1 scores are higher than relation-level accuracy scores, which indicates that LLMs can identify potential connections between entities but often fail to accurately predict the specific relationships.

For models from the same family with different sizes, larger LLMs tend to generate scientific hypotheses more likely to be truthful. This can be attributed to two main factors. First, larger LLMs generally perform better because they can store and leverage more knowledge in their parameters, as shown by the results of parametric knowledge-only setting. Second, LLMs of different sizes have diverse capabilities to process external knowledge for hypothesis generation. For example, GPT-4o-mini shows a modest 1.14% accuracy improvement when augmented with KG and literature, whereas GPT-4o achieves a more substantial 5.14% increase under the same conditions. This suggests that larger LLMs can better utilize additional context to reason about truthful scientific hypotheses. Similar trends are observed when comparing Llama-3.1-8B and Llama-3.1-70B. Interestingly, smaller models, such as Llama-3.1-8B, sometimes experience decreased performance when information from KG and literature is introduced. This degradation may stem from challenges in effectively integrating internal and external information, which can disrupt the model's reasoning processes.

Performance differences are also observed across the three relation types: "Chemical & Gene", "Disease & Gene" and "Gene & Gene". Notably, all larger models, including GPT4o, GPT-4o-mini, and Llama-3.1-70B, tend to perform better on "Chemical & Gene" tasks than on the other two types. This trend suggests that the "Chemical & Gene" task may be more aligned with the pre-trained knowledge or reasoning capabilities of these models. In contrast, the smaller Llama-3.1-8B shows a more inconsistent pattern, with performance varying across tasks and settings, likely reflecting its more limited parametric capacity and reasoning abilities. These variations in performance across relation types may be attributed to differences in training data distributions or the complexity of the relation types themselves. The relatively stronger performance on the "Chemical & Gene" task highlights potential domain-specific biases or strengths in the LLMs, offering insights into their suitability for targeted applications in real-world scientific discovery.

4.3 Hallucination Detection on LLM-generated Hypotheses

To assess the groundedness of the generated hypotheses, we evaluated their rationales using KnowHD under various knowledge settings. KnowHD measures how well a hypothesis is supported by structured knowledge (KG), unstructured knowledge (literature), or both combined. The groundedness evaluation results for hypotheses generated by GPT-4o-mini are presented in Table 3.

The results demonstrate distinct contributions of KG and literature to grounding hypotheses. For example, KnowHD with the literature as the support knowledge base can verify 76.30% claims in the rationales of literature-augmented 'Chemical & Gene" hypotheses. However, the hallucination detector can hardly verify the rationale generated based on adding KG information to parametric knowledge with only 51.08% of the claims being grounded. Combining KG and literature yields the highest groundedness scores, effectively leveraging the complementary strengths of both sources to identify grounded claims and detect hallucinations.

To further explore the relationship between hallucination and truthfulness, Figure 3 examines mean accuracy as a function of groundedness scores. Hypotheses were grouped based on their groundedness scores, and the average accuracy for each group was calculated. The figure reveals a positive correlation between groundedness scores and hypothesis truthfulness. As groundedness scores increase, the likelihood of the hypothesis being truthful also increases. For example, GPT-4o-mini achieves a mean accuracy of 60.96% on "Chemical & Gene" tasks under the combined KG + Literature setting, but this rises to 72.77% for hypotheses with groundedness scores above 80%. These findings underscore the potential of KnowHD to identify hypotheses with a higher probability of being truthful, particularly in contexts enriched with external knowledge.

4.4 Improving Generation of Truthful Hypotheses with KnowHD

To validate the utility of KnowHD on enhancing hypothesis generation, we prompted LLMs to generate five candidate hypotheses for each input and selected the one with the highest groundedness score as the final output. This approach was compared to two baselines: the greedy search method, where the hypothesis is generated using greedy next-token selection by the LLM, and the self-consistency method [Wang et al., 2022], which selects hypotheses based on majority voting across multiple predictions.

As shown in Figure 4, groundedness-based hypothesis selection generally outperforms both the greedy search and majority-voting methods across most knowledge settings. In the parametric knowledge-only setting, the majority-voting method achieves slightly higher accuracy (61.86%) compared to groundedness-based selection (59.83%). However, as external knowledge is introduced, groundedness-based selection demonstrates consistent improvements over both baselines. For example, in the combined parametric + KG + Literature setting, GPT-4o-mini achieves an average accuracy of 63.44% when groundedness-based selection is used, approaching the performance of the larger GPT-4o model.

These results highlight the effectiveness of groundedness scores in scenarios where external knowledge is incorporated, as they help identify hypotheses that are more likely to be truthful. By detecting hallucinations in reasoning steps and focusing on grounded hypotheses, KnowHD provides a robust mechanism for enhancing the reliability and truthfulness of LLM-generated scientific hypotheses.

5 Human Study on Open-ended Tasks

To further assess the generalizability of KnowHD's effectiveness in selecting truthful hypotheses, we conducted experiments on open-ended hypothesis generation tasks. These tasks were designed to evaluate whether KnowHD could reliably identify hypotheses with a higher likelihood of truthfulness across broader and less structured generation scenarios. For this analysis, we utilized the publicly available hypothesis generation dataset introduced by [Qi et al., 2024], which involves generating free-form hypotheses based on given background information. We selected GPT-4o-mini as the tested LLM and enhanced its hypothesis generation process by incorporating external knowledge from scientific literature and knowledge graphs (KG). The model was prompted to generate five distinct scientific hypotheses for each input. These hypotheses were then evaluated by KnowHD, which assessed their groundedness based on their alignment with both structured (KG) and unstructured (literature) knowledge sources.

To analyze the relationship between groundedness scores and hypothesis truthfulness, we filtered generated hypotheses to create pairs with contrasting groundedness levels. For each input, we identified one hypothesis with the highest groundedness score and another with the lowest. We retained pairs where the higher groundedness score was at 30% greater than the lower score. This filtering resulted in 54 pairs of hypotheses with significant differences in groundedness levels. To validate KnowHD's effectiveness, we involved two domain experts to annotate each pair (80% agreement), selecting the hypothesis they deemed more likely truthful based on the given information. Additionally, GPT-4o was prompted to analyze the same pairs and provide its judgment. Results of this annotation study, summarized in Table 4, report the selection ratio for each group, defined as the proportion of hypotheses in each group identified as more truthful.

The results demonstrate a significant relationship between groundedness scores and the perceived truthfulness of hypotheses. Hypotheses with higher groundedness scores were consistently more likely to be selected as truthful by both human experts and GPT-4o, as indicated by the substantial differences in selection ratios. These findings highlight the utility of KnowHD in distinguishing truthful hypotheses, even in unstructured, open-ended generation tasks. By effectively leveraging groundedness as a criterion, KnowHD provides a robust mechanism for improving the reliability of LLMgenerated hypotheses, reinforcing its potential for facilitating real-world scientific discovery processes.

6 RelatedWork

6.1 Scientific Hypothesis Generation

The use of LLMs for scientific hypothesis generation is a rapidly growing field, leveraging the ability of these models to process and synthesize vast amounts of scientific literature [Qi et al., 2023; Yang et al., 2023; Zhou et al., 2024; Ciuc˘a et al., 2023; Park et al., 2024; Skarlinski et al., 2024; Radensky et al., 2024]. LLMs have been applied in identifying research gaps and generating novel hypotheses, with notable successes in areas such as drug discovery, where generated hypotheses have led to experimentally validated drug combinations [Abdel-Rehim et al., 2024]. Despite these advancements, most existing studies emphasize the novelty and diversity of hypotheses without addressing the critical aspect of truthfulness [Qi et al., 2024; Baek et al., 2024; Wang et al., 2023; Hu et al., 2024; Li et al., 2024; Si et al., 2024]. The prevalent hallucination problem exacerbates this issue, as LLMs often generate hypotheses that appear plausible but lack factual support [Huang et al., 2023]. This gap motivates the development of TruthHypo, a benchmark explicitly designed to assess the ability of LLMs to generate truthful and grounded scientific hypotheses.

6.2 Knowledge Graph Reasoning

Knowledge graph reasoning involves inferring missing facts or relationships within a knowledge graph, with tasks such as link prediction, entity classification, and relation extraction being extensively studied [Lin et al., 2015; Ji et al., 2021; Shu et al., 2024]. Traditional link prediction focuses on predicting edges between entities based on graph structure. These tasks primarily target structured graph completion, emphasizing pattern detection rather than creative reasoning [Liu et al., 2023; Wu et al., 2023; Gu and Krenn, 2024]. TruthHypo introduces a novel benchmark that centers on LLM-driven scientific hypothesis generation, leveraging LLMs' ability to flexibly integrate external knowledge through contextual inputs. Unlike static graph reasoning, TruthHypo evaluates how well LLMs generate grounded and truthful hypotheses. This shift highlights the growing role of LLMs in scientific discovery and bridges the gap between symbolic graph reasoning and natural language creativity.

6.3 Retrieval-augmented Generation

Retrieval-augmented generation (RAG) has emerged as a powerful approach for improving the factual accuracy and relevance of LLM outputs by integrating external knowledge during the generation process. This technique has been applied with literature retrieval, as demonstrated by [Lewis et al., 2020], to dynamically incorporate up-to-date information into model outputs. Retrieval-augmented generation methods enhance the ability of LLMs to ground their outputs in external knowledge, making them particularly valuable in tasks requiring factual accuracy, such as scientific text generation [L´ala et al., 2023; Munikoti et al., 2023]. In addition to literature retrieval, retrieval-augmented generation using knowledge graphs has gained attention for its potential to provide structured, domain-specific knowledge during text generation [Peng et al., 2024; Ma et al., 2024; Wang et al., 2025]. TruthHypo builds on this paradigm by integrating both literature and knowledge graph retrieval to provide a robust evaluation of LLMs' ability to generate truthful scientific hypotheses. This dual approach enables a comprehensive analysis of the role of external knowledge in mitigating hallucinations and ensuring the groundedness of generated hypotheses.

7 Conclusion

We presented TruthHypo, a benchmark for evaluating the ability of LLMs to generate truthful scientific hypotheses, and KnowHD, a framework for detecting hallucinations by assessing groundedness in reasoning. Through extensive evaluation, we highlighted the limitations of existing LLMs and demonstrated that selecting highly grounded hypotheses improves truthfulness. These contributions offer valuable insights for improving the reliability and utility of LLMs in scientific discovery.

Acknowledgements

This work is supported in part by the US National Science Foundation under grants 2217071, 2213700, 2106913, 2008208, and NIH grant 1R01LM014012.