UltraModel: A Modeling Paradigm for Industrial Objects

1 Introduction

In the industrial field, modeling techniques play an important role in optimizing the design, operation, and maintenance of industrial systems [Leng et al., 2021; Zhang et al., 2024]. The process of using various modeling techniques to abstract real-world industrial objects into models can be referred to modeling for industrial objects (MIO) tasks. Although industrial objects are diverse, they can be broadly categorized into two main types from modeling perspective. The first type is tangible industrial equipment from the real world, such as the distillation column shown in Fig. 1. This type of industrial objects represent the most important and common targets in the industrial field and the models developed for such objects are often called digital surrogates within the context of digital twins [Shao and Kibira, 2018]. The second type comprises certain abstract variables within industrial processes. These variables are difficult to measure directly through sensors but are crucial to the overall process. Therefore, it is necessary to develop a model to predict them as accurately as possible. Additionally, as Industry 4.0 unfolds, industries face increasing pressure to improve efficiency, reduce costs, and mitigate environmental harm [Borowski, 2021], making the development of accurate and robust models more critical than ever 1 .

∗Corresponding author

Many previous works on modeling for industrial objects have focused on the development of mechanistic models. These models leverage domain knowledge and industryspecific mechanisms to rigorously represent industrial processes [Velten et al., 2024]. However, mechanistic models typically come with high modeling costs and significant computational efforts [McBride et al., 2020; Elsheikh et al., 2023]. On the other hand, machine learning (ML) techniques, especially deep learning (DL), are advancing at an unprecedented pace [Sarker, 2021; Sharifani and Amini, 2023]. Data-driven models have the potential to break free from the constraints of mechanistic approaches, which are highly dependent on prior knowledge and unavoidable assumptions, allowing for more flexible and adaptable modeling for industrial objects. Nevertheless, current efforts to model industrial objects using machine learning or deep learning techniques still face two key limitations.

1 Appendix is available at https://github.com/SpriteAndMango/ UltraModel/blob/main/Appendix.pdf

First, existing research focuses predominantly on one specific industrial object. For instance, Chen and Ierapetritou applied artificial neural networks (ANNs) and support vector regression (SVR) in a hybrid modeling framework to model the Continuous Stirred Tank Reactor (CSTR) [Chen and Ierapetritou, 2020]. Similarly, Li et al. used a surrogate model based on deep convolutional neural networks (CNNs) to model the degradation process of bandsaw blades [Li et al., 2020]. Liu et al. designed a fault diagnosis model based on CNNs for electric machine [Liu et al., 2016]. Su et al. developed a model for predicting the critical properties of chemicals using deep neural networks (DNNs) and long-short-term Memory (LSTM) [Su et al., 2019]. All the aforementioned works share a common limitation: the proposed modeling approaches are restricted to the specific object described in their respective studies. When the modeling object changes, these models fail to adapt effectively, significantly undermining the generalizability of the proposed approaches.

Second, existing research fails to fully consider the latent relationships among variables in industrial data, which limits the model's ability to leverage the data and further results in suboptimal performance [Liu et al., 2016; Chen and Ierapetritou, 2020; Li et al., 2020]. Since many industrial objects can be modeled using the aforementioned mechanistic modeling approach, such mechanistic models can be viewed as a set of equations or functions that represent the behavior of industrial objects. For this type of industrial object, there are clear functional mapping relationships between different variables. This functional mapping is one type of the latent relationships among variables. In addition, there are variables that, while lacking a clear functional mapping relationship, are interdependent and coupled. Such coupled relationships can also be regarded as a form of latent relationships. Therefore, this characteristic distinguishes industrial data for MIO tasks from other datasets. To fully utilize such data, it is crucial to grasp their unique characteristics, thereby maximizing their potential.

To address these limitations, in this paper, we propose an effective modeling paradigm called UltraModel. UltraModel is dedicated to providing a generalizable modeling pipeline capable of solving the majority of MIO tasks. Specifically, UltraModel consists of three main components: twin model graph (TwG) module, multi-scale feature abstraction (MSFA) module and spatial attention-based feature fusion (SAFF) module. The TwG module defines the method for graph construction and, based on the constructed graph, employs graph convolution to generate high-dimensional representations for each node. Furthermore, to fully utilize the latent relationships among variables, the MSFAmodule ingeniously leverages the adjacency matrix of TwG to aggregate information for each node. It then employs a CNN-based network to perform multi-scale feature abstraction on these nodes. Subsequently, the SAFF module fuses the multi-scale features of each node using spatial attention mechanism, resulting in fused representations. Finally, UltraModel generates the model's output by processing the fused representations through a feedforward network. We highlight the contributions of this work as follows:

• To the best of our knowledge, we are the first to develop a modeling paradigm that can effectively address the challenge of modeling different types of industrial object.
• We propose the TwG module, which enables flexible graph construction that dynamically adapts to various industrial objects and task scenarios, thereby enhancing the universality of our approach.
• Building upon TwG, we propose the MSFA and SAFF modules, which complement each other in performing multi-scale feature abstraction and fusion, enabling a thorough exploration of the latent relationships between variables.
• We conduct extensive experiments on two distinct industrial objects, demonstrating that UltraModel outperforms baseline methods and validates its effectiveness for MIO tasks.

2 Preliminary

MIO tasks can be summarized as the design of models capable of predicting outputs for real-world industrial objects based on their inputs. However, for different industrial objects, the meaning of the inputs and what needs to be achieved in the end are different. For tangible industrial equipment, the inputs correspond to the operational variables in the physical world, such as temperature, pressure, the rate of material inflow and outflow, and other variables that can be directly controlled by operators [Zhu and Zhao, 2022]. The ultimate goal in this case is to develop a digital twin model that accurately simulates the behavior of the equipment under identical operational variables. For abstract variables in industrial processes, the input data typically consist of other variables that are closely related and easy to obtain. There may or may not be a clear functional mapping between these input variables and the abstract variables to be predicted. Ultimately, the goal is to develop a model capable of accurately predicting the values of the target abstract variables using these accessible variables. For simplicity, we collectively refer to the models developed for industrial objects as twin models.

The purpose of this study is to propose an effective modeling paradigm to develop twin models for different types of industrial objects. Thus, this work can be summarized as follows:

where, T represents the industrial object to be modeled, OV denotes operational variables used for modeling industrial equipment, EV refers to easily accessible variables used for modeling abstract variables, and TWIN(·) is the twin model we aim to develop.

3 ProposedMethod

3.1 Overview

Fig. 2 illustrates the framework of the proposed UltraModel, which consists of three main components: TwG module, MSFA module, and SAFF module. The TwG module involves two steps. First, it constructs a graph G for the twin model based on functional mapping or coupled relationships among the input variables. Then, it performs graph convolution on the constructed graph G to obtain highdimensional representations X′ for each node. The MSFA module also consists of two steps. First, it aggregates the high-dimensional representations obtained through the graph convolution operation using the adjacency matrix E derived from the TwG module, producing the input Z for the MSFA block. Subsequently, the MSFA block performs multi-scale feature abstraction on Z. The SAFF module is designed to fuse the abstracted multi-scale features into fused representations F. Finally, the fused representations F are processed through a feedforward network to produce the model's output Y .

3.2 TwG: TwinModel Graph

The input data used to model industrial objects are typically sparse and low-dimensional. For example, operational variables such as temperature, pressure, material inflow and outflow are all one-dimensional real numbers. Building an accurate digital twin model of a highly complex industrial object based solely on such sparse and low-dimensional raw input data is extremely difficult. Therefore, appropriate preprocessing of these raw data is a critical step. Here, we adopt a dimensional enhancement strategy.

As mentioned previously, the variables in industrial data used for MIO tasks often exhibit latent relationships, which can be either clear functional mappings or coupled relationships. If we can enhance the dimensionality of the variables while simultaneously incorporating information from other variables that have a latent relationship with the current one, the resulting high-dimensional representations will be more meaningful compared to those obtained solely through dimensional enhancement methods. Motivated by this, we propose the TwG module based on graph neural networks (GNNs).

Graph Construction for the Twin Model

A key characteristic of GNNs is their ability to aggregate information from neighboring nodes while processing the current node [Xu et al., 2018; Du et al., 2021]. This aligns with the aforementioned motivation, prompting us to adopt graph neural network methods to more effectively address MIO tasks. Suppose our input data used to model industrial objects consists of N variables. The values of these variables are one-dimensional real numbers, denoted as xi ∈ R, where i = 1, 2, . . . ,N. Thus, for each sample, we have X = [x1, x2, . . . , xN] and we treat different variables as nodes, represented as V = {v1, v2, . . . , vN}. The construction of graphs for the twin model is determined based on the varying mechanisms of the corresponding industrial objects and can be categorized into three cases: Fully clear mechanisms of industrial object (A1); Partially clear mechanisms of industrial object (A2); Unclear mechanisms of industrial object (A3). More details can be found in Appendix A.1. We denote the graph construction process as G = G(X) in the following. It is worth noting that, in all the aforementioned cases, the weight of any edge connecting nodes is a learnable parameter. Subsequently, we further explore how to leverage GNNs to obtain high-dimensional representations.

Graph-Level Processing

To be general, we begin with the input X ∈ RN×1 . Initially, the input undergoes a linear transformation without bias to map them into a higher-dimensional space, yielding X ∈ RN×D , where D is the dimensionality of the transformed space. Subsequently, we construct the twin model graph G though G = G(X). Then, we apply graph convolution to X based on G. The graph convolution operates as follows:

here, ˜ A = Aw + IN is the adjacency matrix of the graph G with added self-connections. IN is the identity matrix, Aw is the weighted adjacency matrix reflected by E in the graph. ˜Dii = ∑j ˜ Aij, and W (l) ∈ RD×D is a layer-specific trainable weight matrix. σ(·) denotes an activation function, such as the ReLU(·). H(l) ∈ RN×D is the matrix of activations in the lth layer, H(0) = X. The above graph-level processing can be denoted as X′ = GraphConv(X). The resulting X′ = [x′1 , x′2 , . . . , x′N ] denotes the high-dimensional representation obtained from the TwG Module, where x′i ∈ RD , i = 1, 2, . . . , N.

3.3 MSFA: Multi-Scale Feature Abstraction

After obtaining the high-dimensional representation X′ from the TwG module, it is essential to design a network architecture capable of thoroughly exploring and abstracting the highlevel information embedded within these representations.

Constructing Input Data based on the Adjacency Matrix

The ultimate goal of the MSFA module is to use MSFA blocks to perform deeper feature abstraction on the high-dimensional representation X′ generated by the TwG module. However, before this step, X′ needs to be processed to ensure compatibility with the subsequent MSFA blocks. Following the design philosophy of the TwG module, we propose an input data construction method based on the adjacency matrix. The detailed process is as follows.

The inputs to the MSFA module are X′ and the binary adjacency matrix Ab, derived from E. Unlike the weighted adjacency matrix Aw used in graph convolution, Ab is a binary matrix that only indicates connectivity between nodes. Based on Ab, we construct input data for each node according to the following rules. For any node vi, the high-dimensional representations x′j ∈ RD of all nodes vj ∈ N(vi) are stacked along the 0th dimension to form Zi, where i = 1, 2, . . . , N:

if N(vi) contains K nodes, then Zi ∈ RK×D . Subsequently, an additional dimension is added to Zi to facilitate subsequent convolution operations, resulting in Zi ∈ RK×D×1 . A 2D convolutional layer , Conv1(·), with a kernel size of 1 and a channel size of 1 is applied to Zi to obtain the aggregated representation for node i, Zi ∈ R1×D×1 ,

Finally, the aggregated representations of all nodes are concatenated along the channel dimension to form the input Z for the MSFA block, Z ∈ RN×D×1 .

For nodes with unclear mechanisms, as illustrated in Fig. 2, we adopt the strategy of directly using the high-dimensional representations obtained from the TwG module as their final aggregated representations without any additional processing.

Performing Multi-Scale Feature Abstraction

Motivated by advances in computer vision where CNN-based networks are utilized to abstract high-level information from images [Alpay et al., 2024], we designed the MSFA block. The structure of the MSFA block is illustrated in Fig.3. To perform deeper feature abstraction at various receptive fields, we design a parallel, cascaded CNN network with different convolutional kernel sizes and we adopt residual dense block [He et al., 2016; Zhang et al., 2018] into the MSFA. To facilitate deep feature aggregation, we design a recursive feature abstraction mechanism where the input of each layer is constructed by concatenating the outputs of all preceding layers and the original input feature. Formally, the input Z (l) in,k and output Z (l) out,k at layer l are defined as follows, where l = 0, 1, . . . , L:

where CAT (·) denotes the concatenation operation, Conv(·) represents a convolutional transformation, and σ(·) is ReLU(·) activation function. This mechanism allows the model to progressively incorporate multi-level features while preserving the original input information.

3.4 SAFF: Spatial Attention-based Feature Fusion

After processed by MSFA module, we abstract multi-scale, high-level features of the data at different receptive fields. To fully leverage these high-level features, an effective feature fusion network is essential. To this end, we further propose the SAFF module. The structure of the SAFF module is illustrated in Fig.4. Inspired by the channel attention mechanism [Woo et al., 2018], this module takes the high-level features Z (l) in,3 and Z (l) in,1 from the same level but different receptive fields as inputs, and performs feature fusion through a 1 × 1 convolution. Subsequently, based on the logic of channel attention, it generates channel attention scores for the fused features and applies weighting. Finally, the corresponding hierarchical feature F (l) is obtained, where l = 1, 2, . . . ,L, as follows:

where GAP(·), GMP(·) denote global average pooling[ref] and global max pooling, respectively; L(·) refers to a series of linear transformationwith activation functionReLU [Nair and Hinton, 2010]; σ(·) denotes the Sigmoid activation function [Kyurkchiev and Markov, 2015]; P(·) refers to a series of 1 × 1 convolutional operations for feature fusion; ⊙ is the elementwise multiplication operation; and F (l) is the fused hierarchical feature at the lth layer. Finally, the SAFF module performs further fusion of the hierarchical features F (l) from all layers to obtain the final output, F, as follows:

Finally, F undergoes dimensionality reduction and is then fed into a feedforward network to obtain the output Y of the industrial object's twin model, as shown below:

where FFN(·) represents the feedforward network.

4 Experiment

4.1 Experimental Setting

Datasets. We conducted experiments on two fundamentally different types of industrial objects to validate the effectiveness and generalization capabilities of our UltraModel. The first type involves tangible industrial equipment, specifically the distillation column, which is widely used in the chemical industry [Choi et al., 2023]. The second type focuses on abstract variables that are difficult to measure directly yet are critical to industrial processes. Here, we choose the acentric factor, a key parameter that describes the extent to which real gases deviate from ideal gas behavior, playing a significant role in thermodynamics and fluid mechanics [Biswas et al., 2023]. Furthermore, these two industrial objects exhibit a key distinction: the former belongs to industrial objects with unclear mechanisms, while the latter pertains to those with fully clear mechanisms. Consequently, we employ two entirely different types of industrial object to validate the generalizability of the proposed method and leverage these two distinct mechanism scenarios to demonstrate its effectiveness. We refer to the distillation column dataset as DIS-COL and the acentric factor dataset as ACE-FAC. A detailed introduction to these two industrial objects and their datasets can be found in Appendix B.1.

Baselines. To ensure a comprehensive comparison, we selected three different types of models as baselines. These include decision tree-based machine learning models: XGBoost [Chen and Guestrin, 2016] and LightGBM [Ke et al., 2017]; sequence-based deep learning models: Transformer [Vaswani, 2017]; and graph-based deep learning models: GCN [Kipf and Welling, 2016], GAT [Veliˇckovi´c et al., 2017], RADA [Chen et al., 2024], DGDL [Zhu and Zhao, 2022], and TGCN-S [Kong et al., 2023]. Additionally, for the ACE-FAC dataset, where the mechanism is fully clear, we include not only the aforementioned baselines but also its mechanistic model, the Lee-Kesler method [Lee and Kesler, 1975; Boublia et al., 2023]. The results derived from this method serve as the benchmark for model comparison.

Metrics. We use three metrics to evaluate the performance of all models: MAE, RMSE and R 2 [Chicco et al., 2021]. Specifically, on the ACE-FAC dataset, we use the results of its mechanistic model as benchmark. To more intuitively illustrate the performance improvements of other models over the benchmark, we further introduce a metric called Model Improvement Rate (MIR), which is calculated as follows:

where, MAEb and RMSEb represent the MAE and RMSE of the benchmark, respectively.

The experimental setup and implementation details can be found in Appendix B.

4.2 Experimental Results

Performance Comparison on DIS-COL Dataset

This dataset is for modeling real-world industrial equipment, distillation tower. Due to the complexity of the internal mechanisms and the severe coupling of various operational variables affecting the equipment, we regard it as the A3 case and conduct relevant experiments. The experimental results are shown in Table 1. Overall, among the three different types of baselines, the sequence-based deep learning models perform best on this dataset. Compared with the other two types of baselines, it shows a more obvious advantage in all metrics. The second is the decision tree-based machine learning model, and the third is the graph-based deep learning model. Although, Transformer performs best among all baselines, our UltraModel has a 38.04% performance improvement over Transformer. In this dataset, a fully connected strategy is adopted for the connections between nodes, causing graphbased models to aggregate information from all other nodes indiscriminately when generating high-dimensional representations for each node. This prevents them from fully leveraging their advantages, resulting in the performance of the graph-based models not surpassing that of other baselines.

As shown in Fig. 1, the twin model of distillation column constructed in this study is designed to separate a mixture of benzene, toluene, and xylene. To provide a clearer presentation of the experimental results, we calculate the MAE and RMSE between the predicted percentage compositions (y1 , y2 , y3 ) of the three substances and their respective ground truths. The results of all baselines are summarized in Fig. 5, with additional detailed experimental data provided in Appendix C.1.

Performance Comparison on ACE-FAC Dataset

As mentioned earlier, unlike DIS-COL, the mechanisms here are fully clear, enabling us to construct the graph neural network using the A1 method. For graph-based baselines, the graph is no longer fully connected across all nodes; instead, it is constructed based on the functional mapping relationships between different variables. The advantage of this approach lies in the subsequent graph convolution process, where the network aggregates information only from nodes directly related to the current node, rather than adopting an indiscriminate aggregation strategy. Consequently, this type of baseline performs better than other baseline models, with 80% of such models achieving MIR exceeding 80%. This observation further validates the significance of our proposed TwG module from baselines perspective. Among the baselines, RADA performed the best, achieving an MIR of 87.74%. However, our UltraModel still shows a remarkable 58.88% performance improvement over RADA.

4.3 Ablation Studies

We conducted a comprehensive ablation study to assess the individual contributions of each component in our model. The results are summarized in Table 2. To facilitate a more effective comparison between models, we introduced the MIR metric in the ablation study as well. Unlike the previously described setup, the benchmark here refers to the model without the TwG, MSFA, and SAFF modules. Additionally, the MSFA and SAFF modules in our work are designed to complement each other. Therefore, in the ablation studies, we always include or exclude them together. For clarity, we refer to them collectively as the MSFA-SAFF module.

Overall, compared to the case where no modules are included, UltraModel with all modules achieves a 76% performance improvement on the DIS-COL dataset and a 55% improvement on the ACE-FAC dataset. This demonstrates the significant contribution of the TwG and MSFA-SAFF modules to the overall model architecture, as well as their effectiveness. Specifically, when the MSFA-SAFF module is missing, the model is unable to abstract and fuse features in greater depth. Even with the TwG module present, the overall performance of the model remains suboptimal, with improvements of 41% and 40% on the two datasets, respectively. On the other hand, when the MSFA-SAFF module is present but the TwG module is missing, the model can perform in-depth feature abstraction and fusion, resulting in a significant performance improvement. However, without the TwG module, the model cannot generate more meaningful high-dimensional representations prior to feature abstraction, which limits its ability to achieve further breakthroughs. Thus, although the model shows considerable improvement, it does not reach its full potential. In conclusion, the ablation study reveals that the MSFA-SAFF module provides a coarse-grained, substantial performance enhancement, while the TwG module operates at a finer-grained level, complementing the MSFA-SAFF module and enabling further advancements in model performance.

4.4 Visualization

We take the benzene prediction results (y1 ) on the DIS-COL dataset as an example to visualize its predicted values against true values, as shown in Fig. 6. More visualization results can be found in the Appendix C.2. In the figure, the x-axis represents the model's predicted values, and the y-axis represents the real values. The closer the distribution of the model's predictions is to the line y = x, the better the model's performance. As shown in the figure, our UltraModel (represented by red circles) demonstrates a clear advantage over other baselines.

5 Conclusion

In this work, we propose a novel modeling paradigm for various industrial objects (named UltraModel), which is the first attempt to address the challenge of modeling different types of industrial objects. Equipped with the TwG module, our UltraModel can flexibly adapt to various industrial objects with different underlying mechanisms. Furthermore, the collaboration between the MSFA module and the SAFF module enhances the model's feature abstraction and fusion capabilities, significantly improving its data mining capability. We validate UltraModel on two datasets representing distinct industrial objects: the distillation column and acentric factor, both of which are representative but fundamentally different and the results demonstrate UltraModel's superior performance in the MIO task. Hope our research can provide inspiration and assistance to relevant researchers and contribute to filling the gap in the field of industrial modeling.