Multimodal Knowledge Retrieval-Augmented Iterative Alignment for Satellite Commonsense Conversation

1 Introduction

Satellite technology is integral to modern telecommunications, earth observation, and global navigation, with its applications spanning critical areas such as climate change monitoring, disaster management, and the security of global communications networks [Chen et al., 2023a; Lu et al., 2021]. The vast amounts of data generated by satellites offer unprecedented insights into environmental patterns, human activities, and natural phenomena, making them indispensable tools for scientific research, government operations, and commercial ventures [Choi, 2024; Inkollu and Sastry, 2024].

∗ Corresponding author.

With the increasing sophistication of satellite technologies, there is a growing demand for advanced digital platforms and web-based tools capable of efficiently disseminating satellite knowledge to a wider, more diverse audience, including nonexperts [Gallardo, 2024; Li et al., 2024a]. Traditional methods of sharing satellite data, such as technical reports and static datasets, often fail to engage non-specialists and limit the democratization of satellite knowledge. Despite the potential of large language models (LLMs) [Chang et al., 2024; Kasneci et al., 2023] to simplify and popularize satellite knowledge, existing LLMmodels [Thirunavukarasu et al., 2023] face significant limitations when applied to domain-specific data, as shown in Figure 1. Firstly, LLMs struggle with hallucinations, especially in specialized fields like satellite technology, due to a lack of satellite-specific knowledge in their pre-training and tuning phases. This leads to the generation of false or irrelevant information, undermining their reliability. Secondly, the outputs from current models often lack engagement, particularly for non-expert users [Segovia-Dominguez et al., 2021; Gadiraju et al., 2020], as they tend to be monotonous, redundant, and devoid of interactivity.

To avoid the erroneous dissemination of satellite common knowledge by existing LLMs and to enhance the understandability and appeal of the generated results, we propose SatRIA, a multimodal satellite knowledge retrieval-augmented iterative alignment framework for satellite commonsense conversation. including the multi-turn dialogue and visual question-answer tasks. Specifically, we design multi-view retrieval expert knowledge construction that integrates our constructed satellite knowledge graph to reduce token consumption and combines the satellite expert database, satellite rule, and a satellite image database, which allows the model to retrieve and apply domain-specific information during both training and inference. We also design commonsense conversation instructions to make the answers more legible and understandable. To further ensure the reliability of the model's outputs, we design a retrieval-augmented iterative alignment module, that employs a multi-stage evaluation process that continually refines the model's performance by aligning its generated responses with established task-specific standards. This iterative refinement reduces errors and enhances the model's ability to handle complex satellite tasks with greater precision. Finally, we construct satellite commonsense multi-turn dialogue and visual question-answer datasets for satellite commonsense conversation evaluation. Experimental results demonstrate that our approach outperforms existing large language models in answering satellite-commonsense-related questions and provides more comprehensible answers with fewer hallucinations. Our key contributions can be summarized as follows:

• To our best knowledge, we are the first to propose a novel multimodal knowledge retrieval-augmented iterative alignment framework for satellite commonsense conversation to provide more comprehensible answers with fewer hallucinations generation results.
• We design a multi-view retrieval expert knowledge construction to reduce model hallucinations and enhance the interpretability of responses, alongside a retrievalaugmented iterative alignment mechanism that refines outputs through multi-stage evaluations, ensuring taskspecific reliability.
• We construct a satellite knowledge graph, satellite expert database and a multi-modal dataset containing satellite multi-turn dialogue and visual question-answer for training and evaluation across both textual and visual tasks.

2 RelatedWork

Research on applying language models to satellite technology has primarily focused on data classification and anomaly detection [Bondi et al., 2022; Chen et al., 2021; de Witt et al., 2021; Li et al., 2023]. However, these applications have not fully exploited the potential of integrating these models with knowledge graphs to enhance their reasoning capabilities [Han et al., 2020; Lebedev et al., 2019]. While large language models [Zhou et al., 2023; Kalyan, 2024; Huang and Chang, 2023] have shown promise in understanding satellite data, there exists a significant issue of hallucination and a lack of interpretability in their understanding of satellite common knowledge. In addition, we supplement the related work on Multi-Modal LLMs and Domain Large Language Models in the Appendix A.

3 Preliminaries

Satellite commonsense conversation consists of two different tasks - satellite commonsense multi-turn dialogue (SatDiag) and satellite commonsense visual question-answer (SatVQA).

Definition 1 (Satellite Commonsense Multi-Turn Dialogue (SatDiag)). Given a textual query q represented as a set of words, the objective of the SatDiag task is to generate a response r that is coherent and relevant to the context c. The task can be defined as a function fSatDiag : (q, c) → r, where q ∈ Q is a textual query from the set of all possible queries Q, c ∈ C is the context from the set of all possible contexts C, and r ∈ R is the generated response from the set of all possible responses R.

Definition 2 (Satellite Commonsense Visual Question-Answer (SatVQA)). Given a textual query q and a satellite image i, the task is to generate a detailed textual description to answer the question q regarding the satellite image i. It is worth noting that satellite images here refer to pictures of the satellite itself rather than images taken by the satellite. The task can be defined as a function fSatVQA : (q, i) → d, where q ∈ Q is a textual query from the set of all possible queries Q, i ∈ I is a satellite image from the set of all possible satellite images I, and d ∈ D is the generated description from the set of all possible descriptions D.

4 Sat-RIA Framework

LLMs are prone to hallucination, especially in domain-specific fields where professional terminology may be confused or misinterpreted. To address this, we propose a satellite commonsense multimodal knowledge retrieval-augmented iterative alignment framework (Sat-RIA) that not only inputs the query q but also retrieves relevant information from the satellite database s, related parameter satellite design rules r, satellite images i, and the knowledge graph G, as shown in Figure 2. This retrieval-augmented iterative alignment mechanism reduces errors and enhances the model's ability to handle complex satellite tasks with greater precision into the Sat-RIA, enhancing response accuracy and minimizing hallucinations.

4.1 Multi-ViewRetrieval Knowledge Construction

To effectively support the satellite-specific tasks of Sat-RIA, we propose the multi-view retrieval knowledge construction. It integrates data from various sources, including satellite databases, expert knowledge, and visual datasets, to ensure a robust and contextually rich knowledge base. The multi-view retrieval knowledge construction enables the model to dynamically retrieve relevant information across multiple modalities, such as text, images, and structured satellite data, ensuring accurate and informed outputs.

Satellite Expert Database Construction. To address the limitations of existing satellite datasets, we propose the construction of an extensive satellite expert database. Initially, data were collected from Nanosats.eu, which provides information on 2,064 satellites. However, recognizing the limitations of this dataset, we expanded the database by utilizing Google's API to query over 7,000 satellites. The data were further enriched with key parameters from curated expert satellite sources to ensure comprehensive data quality. To resolve discrepancies between multiple data sources, we employ a weighted voting mechanism, defined as follows:

where P ∗ represents the final parameter value, vi denotes the value from the i-th source, and wi corresponds to the weight assigned to that source based on its reliability. This mechanism generates a high-quality, comprehensive satellite database for LLM usage.

Satellite Rule Construction. We propose the design of a Satellite Rule Collection to integrate domain-specific expert knowledge and ensure accurate reasoning in satellite applications. This collection includes a Satellite Design Rules Library, containing 41 guidelines on structural integrity, thermal control, power management, and communication systems, alongside a Satellite Design Formulas Library, comprising 48 key formulas related to orbital mechanics and thermal dynamics (e.g., Kepler's equations). To ensure the robustness and quality of these rules, a validation mechanism was developed, integrating both physical and engineering constraints. The consistency-checking algorithm cross-validates each rule within the Satellite Design Rules Library against these constraints, with the validation process governed by the following scoring function:

where ri denotes the satellite design rule under evaluation, and ck represents the k-th physical or engineering constraint. The function F (ri, ck) outputs a binary value indicating whether the rule ri satisfies constraint ck. A score closer to 1 signifies higher compliance with essential constraints. Additionally, these rules are verified against real-world satellite data, enhancing both their reliability and applicability.

Satellite Knowledge Graph Construction To systematically organize satellite data, we propose the construction of a comprehensive Satellite Knowledge Graph (more details in Appendix B). This knowledge graph G = (V, E) integrates structured data, such as design parameters, mission objectives, operational status, and historical records. The graph comprises nodes V (e.g., satellite, launch vehicle) and edges E (e.g., launched by, operated by). To ensure the integrity and accuracy of the knowledge graph, we propose a consistencycheck mechanism that validates the relationships and data within the graph using multi-relational embeddings. The relationships between entities vk via relationship ej are validated with the following scoring function:

It ensures that relationships are consistent and represent valid connections between satellite components, mission data, and operational parameters. Relationships exceeding a threshold of k are flagged for further review.

Satellite Image Database Construction. We propose the construction of a comprehensive satellite image dataset, encompassing visual data on satellites and launch vehicles. The dataset is compiled through two primary sources: the Google Images Crawler, which retrieves the three most relevant images for each entity using image recognition algorithms, and the Nanosats.eu Crawler, which extracts high-quality images from the Nanosats.eu website, including detailed specifications and mission descriptions. To ensure the relevance and integrity of the image data, we introduce a cross-validation algorithm that evaluates image quality using the Structural Similarity Index (SSI) [Wang et al., 2003] between retrieved images and reference images from trusted databases:

where µx and µy are the means of the reference and retrieved images, σ2x and σ2 y are the variances, and σxy is the covariance between the images. By maximizing the SSI, we ensure that the retrieved satellite images are of high quality and relevant for downstream tasks, such as satellite image analysis.

4.2 Commonsense Conversation Instruction

To ensure comprehensive and robust handling of satelliterelated tasks, we design and implement a multi-faceted approach that integrates both dialogue and visual questionanswer tasks. It allows Sat-RIA to effectively process and generate high-quality responses, tailored to both experts and non-experts in the satellite domain.

SatDiag Instruction. We design a prompt-based satellite multi-turn dialogue system that facilitates natural and coherent interactions between users and Sat-RIA. The key to effective dialogue generation lies in the precise design of prompts, ensuring each interaction targets specific satellite-related knowledge. To elicit accurate satellite-related information, we design prompts for question-answering tasks that guide the model towards retrieving specific knowledge from the satellite knowledge graph. This enables the model to generate factually accurate and contextually relevant responses. A typical prompt for question answering would be:

"Given the specific context provided by the retrieval knowledge [knowledge], how would you address the following query: [query]?"

We next design prompts for multi-turn dialogues to simulate realistic, ongoing conversations about satellite technology. These dialogues allow the model to maintain a coherent flow while addressing multiple aspects of satellite operations or research in a single conversation. A sample prompt for this type of dialogue might be:

"Continuing from the previous discussion on satellite technology, how would you address the following point: [next topic]?"

This design ensures that the model can engage in up to 10 exchanges per satellite, offering a dynamic and engaging dialogue experience. To facilitate the accessibility and comprehension of satellite-related concepts for a diverse audience, we propose a series of commonsense conversation instruction. These instructions focus on enhancing the clarity and engagement of model-generated responses, ensuring that complex ideas are communicated effectively.

1. Tabular Format: Present responses in tabular form to improve clarity and comprehension.
2. Simplified Language: Use simple language and avoid technical jargon to ensure that technical terms are easily understood.
3. Use of Analogies: Incorporate everyday analogies to simplify complex satellite concepts.
4. Storytelling: Engage users with narratives, either real or fictional, to illustrate satellite technologies or discoveries.
5. Proof Links: Provide references to authoritative literature or official sources to enhance verification and credibility.
6. Highlighting Key Points: Highlight critical information with bold text or larger fonts, and incorporate engaging elements like emoticons.
7. Fine-Grained Perception: Identify detailed characteristics, such as color, material, and shape, for a more comprehensive understanding.

By structuring responses in this way, Sat-RIA is designed to effectively communicate complex satellite knowledge, making it more accessible and engaging for a broad audience.

SatVQA Instruction. We propose a detailed multi-step process for constructing satellite visual question-answer to ensure diversity, accuracy, and depth in image interpretation. The construction of this dataset involves collecting a diverse range of satellite images and using advanced language models to generate comprehensive descriptions. In the first step, we use GPT-4V to generate detailed descriptions for each satellite image. These descriptions are crafted using a variety of prompts that focus on different visual and contextual aspects of the image. For example, a prompt might ask the model to describe the visual experience evoked by the image, considering factors such as style, theme, and setting:

"Please observe and describe the experience or feelings elicited by this picture, discussing aspects such as style, theme, setting, mood, and quality."

We further propose a draft creation process wherein the generated descriptions are consolidated with the original descriptions provided on satellite-related websites. This is achieved by instructing GPT-4V to merge information from multiple sources, ensuring consistency and accuracy across descriptions. The prompt used for this task is as follows:

"You are a text information integration expert. Merge the information from two texts describing an image from different perspectives into one comprehensive and detailed description. Retain as much valid information from both texts as possible. Ensure the integrated text is accurate and consistent with the original descriptions."

To ensure accuracy and comprehensiveness, the initial drafts are reviewed by multiple human annotators. These experts refine the descriptions by adding additional details and correcting any errors in multiple rounds of verification, ensuring that the final text is both precise and exhaustive. The final dataset includes 1-5 description dialogues for each satellite image, ensuring depth and variability. Finally, we propose a structured methodology for generating detailed visual question-answers. These descriptions aim to provide enough detail to allow individuals unfamiliar with the image to accurately reconstruct its content based solely on the description.

1. Accuracy: Ensure that descriptions are detailed, factually correct, and free frommisinformation. Provide specifics such as object names (e.g., satellite names) and their attributes (e.g., color, material, shape), and summarize key points in a table for clarity.
2. Question Splitting: For complex queries, divide the description into sub-questions, ensuring that each is addressed thoroughly.
3. Proof Links: Include links to authoritative scientific sources and official websites to ensure the credibility of the description.
4. Highlighting Key Points: Use bold text or larger fonts to emphasize key information, and incorporate engaging elements like emoticons where appropriate.
5. Length: Ensure each visual question-answer is at least 300 words in length, providing sufficient detail for a comprehensive understanding.

4.3 Retrieval-Augmented Iterative Alignment

Retrieval. To optimize response accuracy, we propose a retrieval mechanism that dynamically fetches relevant information from the satellite knowledge graph. The relevance between the query Q and the knowledge graph nodes K is computed using the following function:

where Q is the query vector, K represents nodes in the knowledge graph, and Sim(Q,K) calculates the similarity between the query and the nodes using the cosine similarity function. W(K,Q) is a weighting function that quantifies node relevance to the query, and Imp(K, G) reflects the node's importance within the graph based on its connectivity and strategic position. During inference, these scores are used to retrieve the most pertinent information, ensuring contextually aligned and well-informed responses.

Iterative Alignment. We propose a deconfounded strategy to generate alignment responses during training, allowing the model to progressively align its outputs with the expected form. The model generates multiple candidate responses {y1, y2, . . . , yn} using sampling with different random seeds, while keeping the input x and decoding parameters constant. We also propose an evaluation mechanism based on large locally deployed models (e.g., LLaMa), which use five evaluation criteria to assess each candidate's response. The best response is selected based on a comprehensive score, thereby allowing the model to produce more refined outputs during training. To reduce the cost of data collection at each optimization step, we propose an iterative alignment approach. At each iteration, N multi-modal instructions are selected, and the deconfounded strategy is used to generate n candidate responses with the current model Mi. Each response is evaluated using a locally deployed model L, and the resulting alignment data Di are used to further optimize the model, resulting in Mi+1, which forms the basis for the next iteration. This iterative process continuously improves the model's performance, particularly for multi-modal tasks, while minimizing data collection costs. We implement a weighted optimization approach for alignment, minimizing the discrepancy between the model's output and the expected output with the following formula:

where Mi is the model at iteration i, η is the learning rate, yj is the j-th candidate response, wj is the weight assigned to yj based on its comprehensive score S(yj), and L (yj , x,Mi) is the loss function that measures the discrepancy between yj and the expected output. The comprehensive score S(yj) for each response is computed as:

where ck(yj) represents score for the k-th evaluation criterion, which includes m metrics such as consistency, correctness, context relevance, detail orientation, and keyword accuracy.

5 Experiments

5.1 Evaluation Datasets

To evaluate our models on satellite commonsense conversation, we construct two datasets: one for satellite multi-turn dialogues (SatDiag) and one for satellite visual questionanswering (SatVQA) (more details in Appendix C). The SatDiag dataset includes 2,000 dialogues focused on satellite operations and related knowledge, covering topics like satellite design, formulas, and parameters. Each dialogue is annotated with intent, key entities, and contextual information to assess contextual understanding and response accuracy. The SatVQA dataset consists of 2,000 labeled examples describing satellite images from various scenarios, including components, configurations, and operations. Annotations highlight the primary subjects, notable features, and context, enabling evaluation of multi-modal understanding.

5.2 EvaluationMetrics

To evaluate model performance on SatVQA and SatDiag datasets, we use both traditional and custom metrics (more details in Appendix D). Traditional metrics include BLEU, METEOR, GLEU, and CIDEr, which assess the quality, coherence, relevance, and accuracy of generated text. Custom metrics, evaluated by ChatGPT-4v, measure four dimensions: 1) Consistency: Ensures the response is free from contradictions. 2) Context: Assesses how well the response fits the broader context. 3) Correctness: Evaluates the factual accuracy of the response. 4) Detail: Measures the richness and specificity of the details provided. Each criterion is scored from 0 to 10, with higher scores indicating better performance. Additionally, we evaluate keyword hits using recall, precision, and F1 scores to assess the model's ability to incorporate relevant satellite-related terms.

5.3 ComparisonMethods

We compare our model with several state-of-the-art LLM baselines (more details in Appendix E): 1) InternVL 2 [Chen et al., 2023b] is a multi-modal model with 8B parameters, excelling in visual question-answer tasks but slightly weaker in dialogue generation. 2) LLaVa 1.6 [Liu et al., 2023b; Liu et al., 2023a] is a 7B-parameter model focused on finegrained vision-language alignment, performing well in visual question-answer. 3) Deepseek-VL [Lu et al., 2024] uses 7B parameters for complex image and dialogue tasks, excelling in detail-rich scenes. 4) Yi-VL [Young et al., 2024] is a 6Bparameter model optimized for fast dialogue generation with efficient handling of vision language tasks.

5.4 Implementation Details

The Sat-RIA was trained using the PyTorch framework, leveraging the computational power of NVIDIA GPUs to expedite the training process. we use InternVL 2 8B [Chen et al., 2023b] as the foundational LLM provided a strong baseline for our fine-tuning efforts. We use a total batch size of 1 throughout the training process. The AdamW [Loshchilov and Hutter, 2019] optimizer is applied with a cosine learning rate decay and a warm-up period. In the training stage, every alignment epoch number is 1 with a learning rate of 1 × 10-5

and a warmup ratio of 0.05. Hyperparameters were fine-tuned iteratively based on the performance metrics observed during validation. We have trained our model through the method of full parameter fine-tuning, using a 2xA800 80G machine, and All experiments were conducted on the same machine. For alignment response evaluation, we use the locally deployed LLaMa3 8B model (more details in Appendix F).

5.5 Main Results

To verify the effectiveness of our model, we report the overall average results in Table 1. We experimented with the traditional metrics. From the table, we can observe that: 1) Our model outperforms all others across metrics in both the Satellite Multi-turn Dialogue and Satellite Visual Question-Answer tasks, indicating its superiority in generating accurate and relevant satellite content. 2) In Satellite Multi-turn Dialogue, our model achieves the highest BLEU score of 14.86, well above InternVL 2's 8.27, demonstrating its ability to generate fluent dialogues through satellite expert data and retrieval. 3) Our model's METEOR score of 36.18 for Satellite Multi-turn Dialogue reflects its strength in generating semantically meaningful responses. 4) For Satellite Visual Question-Answer, our model leads with a BLEU score of 17.64, showcasing its capability to produce detailed descriptions. 5) It also excels in METEOR, GLEU, and CIDEr metrics, with scores of 44.41, 21.60, and 10.54, respectively, underlines the robustness and effectiveness of our approach in handling diverse satelliterelated information generation tasks. 6) Compared to baseline models like InternVL 2 and LLaVa 1.6, our model consistently achieves higher scores, confirming the effectiveness of our proposed architecture. All the observations demonstrate the effectiveness of the Sat-RIA framework.

Alignment

Methods Alignment Methods

Figure 3: Variant experiments on evaluating keyword hits.

5.6 Discussion for Model Variants

To investigate the effectiveness of each module in Sat-RIA, we conducted a series of variant experiments, with the results presented in Tables 2 and Figure 3. We evaluated performance using two different types of metrics. In Table 2, we assessed consistency, context, correctness, and detail orientation, as scored by GPT-4V. In Figure 3, we evaluated performance based on Recall, Precision, and F1, which calculate the probability of keyword hits, i.e., the extent to which the generated results cover the keywords.

From Table 2 and Figure 3, we can observe that: 1) Removing the Satellite Expert Database leads to significant performance drops in both tasks. Consistency and context scores fall to 2.78 and 3.00 in Satellite Multi-turn Dialogue, and recall and F1 drop to 0.32 and 0.33, highlighting the importance of expert knowledge. 2) Removing the Retrieval Mechanism results in the lowest scores across most categories. Consistency and correctness for Satellite Multi-turn Dialogue drop to 2.04 and 2.22, and recall and precision for Satellite Visual Question-Answer fall to 0.28 and 0.33, demonstrating the mechanism's crucial role. 3) The variant without the Satellite Rule module exhibits a noticeable decrease in performance,

Rule Expert DB

Mechanism Rule Expert DB

Mechanism particularly in the detail metric for Satellite Visual QuestionAnswer in Table 2, dropping from 6.61 to 6.16. In Figure 3, the F1 score for Satellite Visual Question-Answer drops from 0.64 to 0.61. This suggests that predefined rules are essential for maintaining the detail and accuracy of generated descriptions. 4) Removing Iterative Alignment lowers performance in both tasks. For Satellite Multi-turn Dialogue, detail drops to 7.65, and F1 drops to 0.50. This demonstrates the importance of iterative alignment in refining content and improving relevance and accuracy. 5) The full model (Ours) consistently achieves the highest scores across all metrics, confirming that combining all modules results in the best performance for both tasks. All these observations demonstrate the effectiveness of each component in our model.

5.7 Analysis on SatDiag

We present examples of results generated by our approach compared to other methods, as shown in Figure 4 (a). Results generated using ChatGPT-4o exhibit hallucinations. Additionally, incorporating our designed commonsense conversation instruction makes the generated results more readable and comprehensible. However, due to the inherent hallucinations in LLMs, the provided reference links either do not exist or do not match the answers. Results generated using the LLaMa-8B model not only exhibit hallucinations but also fail to understand questions in the satellite domain accurately. Incorporating commonsense conversation instruction in this model does not yield satisfactory results either. Our model can generate easily understandable results by integrating satellite knowledge, thereby reducing hallucinations. Moreover, the provided references completely match the answers. By incorporating a KG, the responses are enriched and more comprehensible.

5.8 Analysis on SatVQA

We further present examples of results on Satellite Visual Question-Answer generated by our approach compared to other methods, as shown in Figure 4(b). Results generated using ChatGPT-4 are accurate but often fail to capture the key points, making it difficult for those unfamiliar with satellites to quickly comprehend the information. Conversely, results from the LLava 7B model are not only inaccurate but also challenging for laypersons to understand. Our model, on the other hand, generates results that include object detection and recognition within the images, along with some characteristics of the targets. Additionally, by presenting the results in a simple, easy-to-understand manner, including the use of emojis, comprehension is significantly improved. We also provide relevant satellite links, enhancing the explanatory power and reliability of the generated results.

6 Conclusion

This paper introduces a multimodal, retrieval-augmented framework for satellite commonsense conversation. Our framework integrates a satellite knowledge graph, expert database, and image database to reduce hallucinations and enhance response interpretability. It also features a commonsense conversation instruction to improve answer clarity. The retrieval-augmented iterative alignment module refines responses through multi-stage evaluations aligned with taskspecific standards. To evaluate our model, we construct satellite multi-turn dialogue and visual question-answer datasets. Experiments on satellite dialogue and VQA datasets show superior performance over existing LLMs, with fewer hallucinations and more comprehensible answers.

Acknowledgments

We thank the anonymous reviewers for their insightful comments and suggestions. The corresponding author is Qian Li. The authors of this paper were supported by the NSFC through grant No.62402054, No.62425203 and No.62032003, and the 76th batch of general grants from China Postdoctoral Science Foundation through grant 2024M760279, and supported by the Postdoctoral Fellowship Program and China Postdoctoral Science Foundation under Grant Number.

Contribution Statement

Qian Li, Xuchen Li, and Zongyu Chang have made equal contributions and are the co-first authors of this paper.