DAVE: A Framework for Assisted Analysis of Document Collections in Knowledge-Intensive Domains

1 Introduction

Digitalization has expanded access to large document collections, making vast amounts of previously inaccessible information available in digital form. To support users find useful information in document collections and analyze their content, different paradigms have been established to account for different information needs, from exploratory search interfaces [Liu et al., 2024], which typically combine search and filtering functionalities (e.g., under the faceted search paradigm), to question answering systems. The latter ones have become more popular with the advent of Large Language Models (LLMs). Supported by Retrieval Augmented Generation (RAG) architectures, LLMs simplify the development of systems that answer questions in natural language on top of specific document collections [Lewis et al., 2020].

Notably, many domains are inherently entity-centric, where factual information is closely linked to entities that define the context and relevance of a document. For instance, in fields like law and healthcare, professionals seek information on case laws, regulations, diseases, and treatments to support precise search and compliance monitoring. As a result, their information needs are strongly entity-driven.

Consequently, a crucial step toward building entity-aware systems is the adoption of Entity Extraction (EE) approaches to identify and classify entity mentions (i.e., Named Entity Recognition [Li et al., 2022]) and identify links across these mentions. These links can be derived indirectly using background Knowledge Bases (KBs) by applying Named Entity Linking (NEL) techniques [Sevgili et al., 2022] (all mentions linked to the same identifier in the KB are deemed to refer to the same entity) or directly by applying co-reference resolution techniques [Logan IV et al., 2021]. The two approaches can also be somehow combined in end-to-end pipelines combining different components [Pozzi et al., 2023a]. The result of these EE extraction techniques can be used to attach entity-level annotations to the documents, supporting downstream applications for document search and filtering, for example, exploiting faceted search or other semantic search interfaces [Tunkelang, 2022]. However, it is worth noting that EE techniques or more sophisticated methods based on these techniques, are being increasingly used also in RAG applications to improve retrieval and answer formulation in chatbots. Flagship examples of these initiatives are Graph RAG approaches [Edge et al., 2024]. Nonetheless, even more lightweight approaches that enrich content and questions with entities have been shown extremely effective in vertical domain [Xu et al., 2024]. Yet, while both these two paradigms, faceted search and chatbots, can take advantage of entities, their integration is, to the best of our knowledge, limited. Even when used to improve RAG techniques behind chatbots, entities remain in the background.

In this paper, we introduce DAVE, a tool for assisted analysis of document collections in knowledge-intensive domains. The tool goal is to support search needs that span across different points of the extractive vs. abstractive spectrum, as discussed in [Worledge et al., 2024]. It features a graphical user interface (GUI) that enables users to visualize, explore, and query documents. Our tool is specifically tailored for domains where entities are first-class citizens in document analysis and provide the main following features: (i) an entitydriven faceted search interface for entity-driven exploration; (ii) a conversational interface supporting complex natural language queries; (iii) seamless integration of faceted search and conversational interaction to refine document sources; (iv) a human-in-the-loop mechanism for refining entity annotations, ensuring corrections are propagated throughout the system.

DAVE is designed as an open system and is released as open source under the Apache-2.0 license. Documentation, source code, and demonstration videos are available on the project GitHub page 1 .

2 FrameworkMain Functionalities

Knowledge-intensive domains feature extensive collections of complex documents where information is highly factual and deeply connected to real-world entities. Moreover, details about a single entity are often distributed across multiple documents within the corpus. Recognizing that entities are the cornerstone of effective document analysis in knowledgeintensive domains, DAVE proposes an entity-centric exploration approach that goes beyond the exploitation of entity mentions, e.g., as an output of a NER algorithm, supporting mentions linked to entity identifiers and clusters of mentions referring to the same entity.

DAVE provides users with a number of functionalities to address the outlined requirements, which are discussed with examples in legal documents exploration (Figure 1):

• Search & Filter: Users can retrieve documents by entering keywords, which is ideal when they have a clear idea of what they are looking for. Additionally, they can refine their search results by applying multiple filters. For instance, users can filter the corpus based on specific individuals and locations. Figure illustrates the Search & Filter functionalities (the blue section), which allow users to retrieve documents by keyword and refine results through entity-based filters. For example, a user reviewing documents from a trial might search for the keyword "appeal". To narrow the focus, they could use the filtering panel on the left, which organizes filters by entity type (e.g., person, location). By selecting "Licio

1 https://github.com/unimib-datAI/DAVE

Gelli" from the list of identified entities, the user can dynamically refine the results, isolating only those documents where his name appears, with both the result set and available filters updating accordingly.

• Explore: Users can browse the full list of entities identified in each document and navigate directly to sections where they are referenced. In the figure, after selecting a document in the Search & Filter interface, it can be explored in detail via the Explore interface (green section). The entire document is shown with highlighted entity mentions based on type, and a list of entities grouped by mention appears on the left. For example, in the figure, the entity "Valerio Fioravanti" is mentioned 4 times. Clicking any mention takes the user directly to the corresponding text span.
• Conversational Question Answering (QA): Natural language queries are supported, allowing users to ask questions about entities and factual information across multiple documents. In the example (yellow section), the user asks the chatbot about the appeal grounds raised by Prosecutor Squadrini in the Musumeci and Belmonte case, and the DAVE chatbot responds with relevant document passages.
• Knowledge Refinement: Users can refine entity clusters, ensuring that corrections are reflected across all system functionalities. In the figure (red section), the user has identified that the entities "Corte d'Assise di Bologna" and "Corte di Assise di Bologna" with their respective mentions are actually equivalent, therefore is collapsing the two entity clusters into one.

DAVE also supports search composability, enabling users to combine Search, Filter, and Conversational Question Answering. By filtering results before querying the RAG-LLM

system, the chatbot works with a more focused document set, improving answer precision and relevance. Additionally, DAVE ensures data privacy through on-premise servers and access controls, crucial for sensitive domains.

By offering this suite of functionalities, DAVE enables precise control and in-depth understanding of large corpora, providing diverse exploration and search capabilities tailored to knowledge-intensive domains and specialized users. To this aim we implemented these functionalities by mixing the following techniques and technologies:

• Entity-centric management. Entity-level annotations are represented in the GATE format [Cunningham, 2002] and stored in a MongoDB database; the DB stores information about the annotations for every entity mentions, entity identifiers, and links between annotations and entity identifiers. The current prototype considers annotations resulting from pipelines that apply algorithms for NER, Named Entity Linking (with links to Wikipedia), NIL Prediction (to identify entities not represented in Wikipedia), and NIL Clustering (to cluster NIL entities and create identifiers for each cluster)[Pozzi et al., 2023a; Pozzi et al., 2023b; Bellandi et al., 2024]. As a result, each entity mention can be linked to an entity identifier, either external (e.g. a Wikipedia URI), or internal (identifying a local cluster). Each cluster is associated with a default surface form used to display the entities in the interface.
• Keyword and Faceted Search: The Search engine uses keyword matching to efficiently retrieve relevant documents and serves as the foundation for several other functionalities. To support the Search & Filter and Explore functionalities, DAVE employs the wellestablished faceted search paradigm. This technique provides filtering facets based on entities, allowing users to refine their search results. By providing a structured way to narrow down results, faceted search enhances the user's ability to explore large corpora. Users can apply filters either across the entire corpus or within a subset of documents for a more granular exploration.
• Human-in-the-Loop (HITL): The Knowledge Consolidation feature follows the HITL paradigm, ensuring continuous user involvement in refining the system. Users can correct and refine annotations and entity clusters, and these corrections are reflected across all system functionalities. This active participation helps the system improve over time, ensuring a pay-as-you-go consolidation of the background data as proposed for similar tasks [De Castilho et al., 2024; De Paoli et al., 2019; Cutrona et al., 2019; Cruz et al., 2016].
• Retrieval Augmented generation (RAG): The Conversational QA functionality is powered by an LLMbased chatbot, implemented through the RAG paradigm. This enables users to ask fact-based queries about entities across multiple documents. While LLMs excel in natural language understanding and zero-shot learning, RAG ensures responses are grounded in retrieved documents, addressing concerns about hallucinations and limited knowledge, which is crucial in certain domains.

3 Applications, Main Contributions and Demonstration

DAVE has been used in prototypes for Italian projects in the legal domain [Batini et al., 2024] with the goal of showing stakeholders AI-powered search functionalities, we have experimented DAVE in i) search on court decision in civil trials [Bellandi et al., 2024], ii) criminal investigations and chat analysis [Pozzi et al., 2025], and iii) analysis of the documentation about the Bologna massacre of August 2, 1980.

In relation with related work, we discuss below the three main novelties that we believe DAVE presents as a system:

• Mixing entity-driven faceted search and conversational assistant. Entity-driven faceted search is a mainstream technology in many knowledge-driven scenarios where documents are analyzed using entity extraction (EE) methods [Guo et al., 2023; Hirsch et al., 2021]. On the other hand, RAG systems on pre-filtered data have been studied, with approaches like agent-based filtering [Poliakov and Shvai, 2024], metadata-based filtering [Chang et al., 2024], and natural language inference [Yoran et al., 2023]. Our work combines faceted search with a RAG system, enabling dynamic, entitydriven document filtering.
• Integrating interactive entity-driven knowledge consolidation in an information exploration interface. Several platforms support text annotations, with Doccano [Nakayama et al., 2018] being widely used; Very recent work has surveyed interactive approaches to improve annotations, minimize user effort, manage annotation teams, support pre-annotated data, and enable customizable task design [De Castilho et al., 2024]. While we haven't fully integrated the advanced annotation quality methods from [Klie et al., 2020], our application is the first to integrate incorporate interactive methods editing into an exploratory search interface, focusing on improving entity clustering, a key challenge in end-to-end EE pipelines.
• Entity-centric RAG prototyping and grounding. Frameworks like LangChain [Chase, 2022] facilitate rapid prototyping and configuration of RAG systems, while tools such as RAGAS [Es et al., 2024] and RAGChecker [Ru et al., 2024] allow for detailed evaluation through a wide range of metrics. However, existing applications do not support the prototyping of highly entity-centric LLM-RAG systems that enable direct analysis of the corpus and its entities to verify factual accuracy, making effective debugging more challenging.

These innovations arise from the need for domain experts to thoroughly analyze and explore documents annotated by entity extraction pipelines, using established, user-friendly search paradigms and allowing experts to improve the system by refining annotations. A first quantitative evaluation where DAVE outputs are compared to outputs of top-tier models is ongoing in the context of a Civil Appeal Proceedings use case [Agazzi et al., 2024].

During the demonstration session, users are guided in exploring a document collection using DAVE's features.

Acknowledgments

The work has received funding from the European Union's Horizon Europe research and innovation programme under grant agreements No. 101189771 (DataPACT) and No. 101070284 (enRichMyData), and the European Community - Next Generation EU under the Italian Ministry of Justice, and the Italian PRIN project Discount Quality for Responsible Data Science (202248FWFS). We also thank "Associazione 2 Agosto 1980" for the useful discussions and their support for using their data in our demo.