NSF-MAP: Neurosymbolic Multimodal Fusion for Robust and Interpretable Anomaly Prediction in Assembly Pipelines

1 Introduction

Anomaly prediction in industrial assembly pipelines is essential for maintaining product quality and operational efficiency. Traditional methods focusing on unimodal data [Chandola and Banerjee, 2009; Kim et al., 2023; Shyalika et al., 2024a] often fall short in complex environments where anomalies are influenced by multiple variables. Multimodal

Video URL:https://shorturl.at/LCdsO

anomaly prediction provides a more comprehensive and accurate prediction mechanism by uncovering hidden patterns and relationships that single-modality methods often overlook [Nedelkoski et al., 2019; Zhao et al., 2020]. This approach is particularly beneficial in manufacturing settings where variables like temperature, speed, and visual inspections are interrelated, necessitating advanced methods to handle multimodal and multivariate data effectively [Zhao et al., 2020; Nedelkoski et al., 2019]. Despite the benefits of multimodal anomaly prediction, challenges such as the extreme scarcity of datasets containing both time-series and image data-where we identified only one (details in Section 3)-along with poor performance metrics, real-time integration difficulties, and limited interpretability persist in manufacturing assembly processes, highlighting the need for more practical, scalable, and interpretable solutions [Shyalika et al., 2024c; Shyalika et al., 2024b; McCormick et al., 2025]

Multimodal fusion has emerged recently as a promising approach for integrating multiple data modalities to enhance the accuracy and robustness of various applications [Zhang et al., 2024; Jaafar and Lachiri, 2023]. This integration leverages the complementary strengths of each modality, enabling a more comprehensive analysis. Decision-level fusion (a.k.a. late fusion) is a fusion technique that combines the decisions of multiple classifiers into a shared decision about any activity that occurred [Roggen et al., 2013]. Here, each classifier independently analyzes features and makes preliminary decisions, which are then aggregated into a fused decision vector to derive the final task decision[Kaya et al., 2017; Meng et al., 2013]. However, since the classifiers operate independently, decision-level fusion often fails to capture implicit semantic associations among the variables, leading to a loss of context, inconsistencies among classifier outputs, and challenges in interpretability and scalability.

Neurosymbolic AI integrates neural network-based techniques with symbolic knowledge-based approaches, offering two key perspectives for understanding this integration: (1) algorithmic-level considerations and (2) application-level considerations in AI systems [Sheth et al., 2023]. This work uses a neurosymbolic AI-based knowledge-infusion for anomaly prediction. Knowledge-infusion allows us to integrate domain knowledge with data-driven approaches, providing a foundation for more robust and interpretable AI models. Building on this concept, we propose NSF-MAP (Neurosymbolic Multimodal Fusion for Robust and Interpretable Anomaly Prediction), a novel framework specifically designed for robust and interpretable anomaly prediction in manufacturing pipelines. This work is part of a broader research initiative focused on the development of SmartPilot, a custom, compact, and neurosymbolic CoPilot for intelligent manufacturing [Shyalika et al., 2025b; Shyalika et al., 2025a]. NSF-MAP is the anomaly prediction agent within SmartPilot's multi-agent architecture, to support robust, real-time decision-making in industrial settings.

• A novel anomaly prediction model for assembly pipelines that leverages neurosymbolic AI-based decision-level fusion combined with transfer learning.
• Two derived datasets, one multimodal and one analog, designed for anomaly prediction in assembly pipelines, developed using domain-specific insights.
• Integration of advanced user-level explainability techniques, including an ontology-based method, to enhance the interpretability of our predictions.

2 RelatedWork

2.1 FusionModels and Transfer Learning Techniques

Recent advancements combine time series and image fusion with transfer learning to improve anomaly detection and prediction across domains. In manufacturing, the MFGAN model [Qu et al., 2024] leverages attention-based autoencoders with GANs on multimodal temporal data for anomaly prediction, while fusion models like ESTARFM [Wang et al., 2014], SRCNN+LSTM [Yang et al., 2021], and DTWCNN [Iwana and Uchida, 2020] enhance spatial, temporal, and sequential learning in remote sensing. In healthcare, integrating EHR with radiographic images improves atypical femur fracture classification [Schilcher et al., 2024]. Transfer learning methods address data scarcity and dynamic conditions by adapting learned representations across tasks in industrial [Yan et al., 2024; Maschler et al., 2021], surveillance [Jayaswal and Dixit, 2021], and web domains [Zhang et al., 2022]. Techniques like Variational Autoencoders, Graph Attention Networks, and fuzzy classifiers further improve detection accuracy [Series, 2024; Lughofer et al., 2022], though challenges remain in domain adaptation, model complexity, real-time processing, and interpretability.

2.2 Neurosymbolic AI inManufacturing

Knowledge graphs have been applied to anomaly detection by embedding structural and attribute features for threat assessment [Moghaddam et al., 2023; Zhuo et al., 2021; Zhao et al., 2022; Wang et al., 2023]. Approaches include cybersecurity knowledge graphs with GNNs [Zhuo et al., 2021; Wang et al., 2023], Neo4j-based power database evaluation [Zhao et al., 2022], hybrid neuro-symbolic methods combining data-driven models with industrial ontologies [Capogrosso et al., 2023], and predictive maintenance for industrial robots [Wang et al., 2023]. However, no existing work leverages neurosymbolic AI-based multimodal fusion for anomaly prediction in manufacturing. Integrating symbolic reasoning with neural models on multimodal data holds potential to improve robustness and interpretability in complex industrial settings.

3 Data Preparation

Time Series Data Preparation We use the publicly available manufacturing dataset [Harik et al., 2024] generated by the Future Factories (FF) Lab at the McNair Aerospace Research Center at the University of South Carolina. This dataset has two versions: analog and multimodal datasets. The dataset consists of measurements using a prototype of rocket assembly pipeline [see Appendix A], which adheres to industrial standards in deploying actuators, control mechanisms, and transducers. The multimodal dataset includes synchronized images captured from two cameras positioned on opposite sides of the testbed, which continuously record the operations. It consists of 166K records throughout the entire runtime of 30 hours with a sampling rate of 1.95 Hz. The images are divided into batches of 1K samples from both camera viewpoints, with each batch having an equivalent JSON file containing synchronized sensor values and their corresponding image paths. In data preparation, data from JSON files were first converted into a tabular format, including the paths to the camera images. The data consists of 285 assembly cycles, at which one cycle represents a complete assembly and disassembly of the rocket. The rocket assembly process at the FF lab is divided into 21 distinct cycle states. Information about these cell cycle states and cycle state-wise anomaly types is not directly available in both the analog and multimodal datasets and we extracted them using a mapping function provided by domain experts. The preprocessed dataset includes measurements, such as sensor variables, conveyor variable frequency, drive temperatures, physical properties of robots (i.e., angles), conveyor workstation statistics, cycle state, cycle count, anomaly types and corresponding image file names from both cameras. In this paper, we introduce the preprocessed multimodal and analog datasets, incorporating this new information. Table 1 presents the statistics of anomalies in the preprocessed multimodal dataset used in the proposed method of this study.

ImageData Preparation We use the derived image dataset from [Prasad et al., 2024]. In this assembly pipeline, rockets and components are visible only during certain cycle states due to the positions of robots, machinery, and cameras. Using domain expertise, images from cycle four and part of cycle nine are filtered to enhance analysis and further cropped to focus on the region of interest.

4 ProblemFormulation

Consider a multimodal dataset M, comprising time series data T and corresponding image data V, collected while assembling a rocket. The dataset consists of n cycles, denoted by C = {c1, c2, . . . , cn}, where each cycle ci for i ∈ {1, 2, . . . , n} represents the complete assembly of a product. Each cycle ci is divided in to 21 distinct states, represented by S = {s1, s2, . . . , s21}. Let Vci,sj = {vci,sj,1, vci,sj,2, . . . , vci,sj,T} denote the image data for cycle ci at state sj. T comprises multivariate sensor readings at various time steps within each cycle state. Let Tci,sj = {tci,sj,1, tci,sj,2, . . . , tci,sj,T}, where tci,sj,k = (tci,sj,k,1, tci,sj,k,2, . . . , tci,sj,k,m, yci,sj,k) represents the multivariate sensor measurements at time step k for cycle ci at state sj . Here, m is the number of sensor variables, tci,sj,k,l denotes the sensor measurement for the l-th sensor variable, and yci,sj,k is the anomaly label associated with time step k. Additionally, let R = {R(1) ,R(2) , . . . ,R(m) } denote set of acceptable sensor ranges, where R(l) represents the acceptable range for the l-th sensor variable. The objective is to develop a model, F, for predicting next sensor values tci,sj,k+1 and anomaly label yci,sj,k in the next time step of the assembly process.

5 NSF-MAPMethod

This section describes the NSF-MAP method (Figure 1), including feature extraction, fusion, and prediction approach.

https://github.com/renjithk4/AssemAI

Then, we elaborate on the formulation of the loss function and the integration of the neurosymbolic AI approach utilized in this framework.

Image and Time Series Feature Extraction

We focus on images captured at states s4 and s9, which are critical for anomaly prediction. The region of interest(rocket parts) for each image Vci,sj is identified using a bounding box Bci,sj = (xmin, ymin, xmax, ymax), which is obtained from the YOLO-FF model [Prasad et al., 2024]. Then, we use a pretrained EfficientNet model GV with the classification layer removed to extract the trained image embeddings fV as defined in equation 1.

For time series data, an autoencoder-decoder network GT is used to extract features into a latent representation fT and to reconstruct the input sequence. The encoder ET (defined in equation 2) extracts features, while the decoder PT (defined in equation 3) reconstructs the sequence.

The overall process of the autoencoder-decoder network GT can be described as in equation 4.

Decision-level Fusion and Prediction

In this approach, we integrate the features extracted from the models GT and GV . The features fV from GV and hT from GT are concatenated to form a unified feature vector z as in equation 5.

This unified feature vector z is then passed through a fully connected network H to obtain the final prediction as in equation 6.

where yci,sj,k+1 represents both the predicted sensor values ˆtci,sj,k+1 and the anomaly label ˆ yci,sj,k+1 at the next time step k+1 for cycle ci at state sj . In all other states except for states s4 and s9, the features hT from the autoencoderdecoder network are used directly for prediction as defined in equation 7.

We illustrate the decision-level fusion and prediction in Figure 1 (Method P1).

Limitations and Transfer Learning Strategy

The end-to-end training of the autoencoder-decoder may lead to overfitting, longer training times, and higher computational costs due to the decoder adapting too closely to specific time series features. To mitigate this, transfer learning [Torrey and Shavlik, 2010; Pan and Yang, 2009] is applied by freezing the encoder ET and training only the decoder RT , optimizing solely θRT . The prediction model H then utilizes features hT extracted by the fixed encoder, with fT generated accordingly. The overall objective function remains unchanged, as illustrated in Figure 1 (Method P2).

Process Ontology for Knowledge-Infusion

We design and develop Dynamic Process Ontology to support knowledge-infused learning and provide user-level explanations for model predictions. The ontology is built on the basis of 21 cycle states S = {s1, s2, . . . , s21}, capturing the temporal component of the assembly process and hence the name Process Ontology. Additionally, the ontology can be dynamically updated as per the experimentation set-up to adjust the expected minimum and maximum values of the sensor, the function of robots in the cycle states, and other information as required (example in Appendix B). The specific attributes of process ontology are as follows: (a) definitions and item specifications of each sensor and equipment, (b) relationships between the sensors and equipment, (c) function and involvement of each sensor and robot with respect to the cycle states, (d) expected (or anomalous) values of sensor variables concerning each cycle state, (e) types of anomalies that could be associated with each cycle state, and (f) sensor values and properties that can be dynamically updated as per changes in the experimental set-up.

While each sensor has possible minimum and maximum values that remain constant throughout the assembly process, the expected normal operating range varies based on the sensor's role in a given cycle state. For example, a potentiometer's expected range on robot-1 varies by cycle state. This necessitates an understanding of the relationships between cycle states, robot involvement, and corresponding sensor readings. Therefore, we gather this domain knowledge from the Process Ontology to guide the model training process. An ontology dynamically integrates sensor ranges by adapting expected values to cycle states, workload, and equipment interactions, enabling real-time reasoning and explainability beyond static thresholds. In this work, we integrated domain-specific knowledge, particularly sensor ranges R = {R(1) ,R(2) , . . . ,R(m) }, to improve the model's performance and enhance its robustness. By incorporating domain constraints into the learning process, the Knowledge Infusion approach ensures that the model generalizes beyond fixed thresholding, reduces false positives and negatives, and effectively reasons over complex multi-sensor dependencies that cannot be captured through simple thresholding alone.

https://github.com/revathyramanan/

Dynamic-Process-Ontology

In model training, the predictions must satisfy:

The penalty P is added in two specific scenarios, as outlined in Equation 8. A penalty is imposed when the model predicts an anomaly even though the sensor values are within the acceptable range, indicating that the model incorrectly identifies an anomaly where none exists. A penalty is also applied when the model predicts normal conditions despite the sensor values being outside the acceptable range, reflecting a failure to detect an anomaly.

where 1 is an indicator function that adds a penalty when the conditions inside it are met.

Loss Function

We employ a Weighted Mean-Squared Error (WMSE) loss function to address the class imbalance and to further differentiate between the different types of anomalies. Let w be the vector of class weights, y be the true values, and p be the predicted values, the loss function can be defined as in equation 9.

The objective is to minimize the WMSE for predicting the next sensor values and anomaly label as in equation 10.

The final predictions yci,sj,k+1 are then fed into a classification layer C, which will filter and classify the anomaly label ˆ yci,sj,k+1 into separate anomaly types (equation 11).

where ˆaci,sj,k+1 represents the predicted anomaly type at the next time step k + 1 for cycle ci at state sj. The full objective function, with the penalty, is as in equation 12.

where wi,j,l are the sample weights, λ is a hyperparameter controlling the strength of the penalty term and P is the penalty function defined above. We illustrate the knowledgeinfusion approach in Figure 1 (Method P3). The application of this ontology for providing user-level explanations is elaborated in the Results section.

6 Experiments

6.1 Common Hyperparameters and Training Setup

The dataset is split using a cycle-wise approach, ensuring 8020% and 60-40% training-testing splits while maintaining the same proportion of normal and anomalous samples in both training and testing sets. For all models, we use WMSE, tuning hyperparameters like epochs(50), batch size(32), and learning rate(0.001). The best model is saved based on validation accuracy. All the model parameters are optimized using the Adam optimizer, and the training process involves a training phase to adjust the model weights and a validation phase to monitor the model's performance on unseen data. During preprocessing, we resize the original images to 224×224 pixels and normalize them using the mean [0.485,0.456,0.406] and standard deviation [0.229,0.224,0.225]. In the following section, we describe the experimentation details of baselines in B1 and B2 and proposed methods in P1, P2, and P3, respectively.

B1:Time Series-based Autoencoder Model

The autoencoder model consists of two main components: an encoder and a decoder. The encoder is a single linear layer that compresses the input data (we use two most important sensor variables and anomaly label) into a lower-dimensional representation with a hidden size of 10, with a ReLU activation function applied to introduce non-linearity. The decoder is a single linear layer that reconstructs the original input from this compressed representation. The autoencoder aims to learn a compact representation of the input data that can effectively capture its essential features. The model optimizes the WMSE between the predicted next-time series values (sensor values and anomaly label) and the current values.

B2:Image-based EfficientNet-B0 model

We implement an EfficientNet-B0 model [Tan and Le, 2019] on the preprocessed images for an anomaly detection task, initially pre-trained on the ImageNet dataset. We adapt the model by modifying the final classification layer to align with the number of classes in our image dataset (five classes).

P1: Decision Level Fusion

(DLF) (Figure 1: P1) The autoencoder model is designed with an encoder that takes an input dimension of 3, a hidden dimension of 64, and a latent dimension of 128. It utilizes the ReLU activation function. The decoder receives the latent dimension of 128 and reconstructs the subsequent time series data with an output dimension of 3, also employing the ReLU activation function. The EfficientNet-B0 model is pretrained and modified by removing its final classification layer. This alteration allows its features to be concatenated with the latent features from the time series before being passed to a fully connected layer with an output dimension of 3. The concatenated output of autoencoder and EfficientNet-B0 is fed to a fully connected layer, which has a dropout rate of 0.5 for regularization. The model aims to predict the next time series data with an output dimension of 3. A ReduceLROnPlateau scheduler adjusts the learning rate based on validation loss, with a patience of 5 epochs. Early stopping is implemented to prevent overfitting, with the best model saved based on the lowest validation loss.

P2: Decision Level Fusion with Transfer Learning

(DLF+TL)(Figure 1: P2) The hyperparameters of autoencoder and EfficientNet-B0, training process, and loss function will be the same as in the P1. The encoder is frozen to disable gradient updates for the encoder parameters.

P3: Enhanced Decision-Level Fusion with Transfer Learning through Neurosymbolic AI

(DLF+TL+KIL) (Figure 1: P3) The hyperparameters, training process, and loss function for the autoencoder and EfficientNet-B0 remain consistent with those in P1. In this proposed approach, we implement a custom loss function to P2 that includes WMSE loss with an additional penalty, as explained in the Methodology section.

7 Results

Table 2 summarizes the results of our experiments on the test set and the ablation studies across different baselines. We evaluate the performance using four metrics: weighted averages of precision, recall, F1-score, and accuracy. The weighted averages are calculated based on the six types of anomaly classes and the normal class. In the 80-20% split, the Decision-Level Fusion model achieves an overall accuracy of 72%, with a precision of 76.05%, a recall of 72%, and an F1-score of 72.03%. With the inclusion of Transfer learning module, the accuracy and the F1 score increased by 39% and 45%, respectively. Similarly, infusion of knowledge on sensors enhanced the accuracy and the F1 score by 47% and 52%, respectively. Figure 2 depicts the performance of detecting various anomaly types and the normal class across various modeling approaches. It can be observed that among all the models, Decision level fusion with transfer learning and knowledge-infused learning approach gives the best results in detecting five out of seven types of classes. Figure 3 presents the comparison of performance metrics for different models under the two training-testing split scenarios (80%- 20% and 60%-40%), demonstrating the effectiveness of the proposed approach against baseline models.

7.1 Process Ontology for User-level Explainability

Dynamic Process Ontology is utilized to provide user-level explanations for the outputs generated by the model. The model predicts the values of the sensors before passing them on to the classification layer. In both scenarios, whether an anomaly is present or not, the variable values predicted by the models can be used to explain the anomaly predicted by the model. For example, if a value for variable-1 is predicted to be 3000 in cycle state 4, where its expected values are supposed to be 6000 to 8000, the anomaly can be Models Models explained as follows: (i) which variable is responsible for the anomaly? (ii) what were the function of the robots during that state? (iii) what are the expected values of that variable in that state? The ontology can also capture misclassification in certain scenarios. If the model predicted the anomaly type to be "NoNose" in cycle state 4, it can be observed from ontology that "NoNose" anomaly happens only from cycle state 8. Similarly, if all the variables are predicted to be within the expected range for a given cycle state and the model predicted still predicted them as anomaly; these misclassifications can be identified by the ontology. To summarize, the ontology serves as a knowledge-infusion and an explainability tool along with serving as a mechanism to identify misclassifications in certain cases if at all those were missed by the model.

Figure 3: Performance of the Proposed Approach and Baselines with Varied Training and Testing Splits. ZTI: Zero Tensor Image

8 Deployment of NSF-MAP

We deploy the NSF-MAP and process ontology at the FF testbed at the McNair Center, University of South Carolina. The deployment strategy of this model consists of two main parts (See deployment setup in Appendix C]). First, it requires establishing connections between the trained model, ontology, and the system. This includes real-time data retrieval through an Open Platform Communications Unified Architecture (OPC-UA) server on the system's Programmable Logic Controller (PLC) and accessing analog data via an OPC client script. Additionally, cameras around the system provide necessary images, accessed using custom Python libraries, Pyplon and ImageCap. Second, the internal pipeline is created to integrate the ontology and NSF-MAP. Data from the OPC-UA server is ingested into a process ontology, running on a local Neo4j server via Docker, to generate user-level explanations. These explanations and the corresponding images and analog data are then fed into NSF-MAP for real-time anomaly prediction. We face several challenges in deployment, which have been explained in Appendix C].

9 Conclusion and FurtherWork

In this study, we derived industry-standard datasets tailored for analog and multimodal assembly processes. We developed a novel methodology for robust and interpretable anomaly prediction using neurosymbolic multimodal fusion. Our findings indicate that the time-series-image fusion model incorporated with a neurosymbolic AI approach offers significant improvements over traditional methods. Future work will explore hybrid architectures that integrate additional modalities, such as text, with advanced interpretability methods for real-time anomaly prediction. We also plan to extend the process ontology by incorporating images aligned via techniques like dynamic time warping, change-point detection, or saliency analysis. To enhance expert understanding, we propose generating abstract representations of causal factors linked to anomalies for a more holistic interpretation of anomalous events.

A Future Factories Setup

Figure 4 includes an image of a rocket assembled by FF Lab.

A visual representation of the lab setup is shown in Figure 5.

B Experiments with the Process Ontology

A snapshot of the ontology for a given cycle state is given in Figure 6.

C Deployment of NSF-MAP and Challenges Faced

Figure 7 illustrates the architecture for real-time deployment of the NSF-MAP model and process ontology onto the manufacturing system. It highlights integrating the trained model with the OPC-UA server for data retrieval and the connection to cameras for image acquisition, enabling seamless real-time predictions on the FF testbed. The inference code, user interface code for deployment, and the demo of deployment are included in the supplementary files.

During the deployment of NSF-MAP in the real-world manufacturing environment at the FF lab, several challenges arise that require careful management. One of the primary concerns is adapting the code to align with the format of the input data stream, which includes both time series and image data, to ensure seamless integration and functionality. Integrating the cameras with the local PC, where the process ontology and NSF-MAP are deployed, presents an additional effort. A significant challenge is the discrepancy in the sampling rates between the OPC server and the data set used for training, necessitating efforts to synchronize these rates. Additionally, the data used for the current deployment were collected eight months prior, which may lead to slight differences in the sensor data ranges. As a result, considerable effort is devoted to adjusting and fine-tuning the code to accommodate the new data stream. The simulation of anomalies presents another challenge, as accurately replicating realworld scenarios proves difficult. Furthermore, the selection of appropriate robots and sensor values for testing, particularly in conjunction with simulated anomalies, is a task that requires close collaboration with domain specialists from the FF lab. Despite these challenges, the combined efforts and expertise of all stakeholders facilitate the successful resolution of these issues, thereby contributing to the advancement and refinement of NSF-MAP's deployment in the manufacturing environment.

Ethical Statement

There are no ethical issues.

Acknowledgments

This work is supported in part by the NSF grant #2119654, "RII Track 2 FEC: Enabling Factory to Factory (F2F) Networking for Future Manufacturing".

Contribution Statement

∗Chathurangi Shyalika and Renjith Prasad contributed equally to this work. †Chathurangi Shyalika is the corresponding author of this work.