A Fast Neural Architecture Search Method for Multi-Modal Classification via Knowledge Sharing

1 Introduction

With the advent of the big data era, the internet generates a vast amount of data in various modalities such as image, text, and sound. Compared to single-modal data, multi-modal data contain more diverse relationships and deeper semantic information. Multi-modal classification refers to the utilization of multi-modal data for understanding and categorizing targets, with its core lying in how to effectively fuse these multimodal information [Liang et al., 2022; Jiang et al., 2023; Han et al., 2022].

∗Corresponding Author

In order to find the optimal network structure for multimodal information fusion, researchers have focused on NAS. NAS-MMC can automatically search for more efficient and well-generalized multi-modal fusion strategies within a vast architectural space, such as DC-NAS [Liang et al., 2024], CSG-NAS [Fu et al., 2024], BM-NAS [Yin et al., 2022], EDF [Liang et al., 2021], and MFAS [P´erez-R´ua et al., 2019]. Based on different search strategies, NAS is mainly divided into methods based on reinforcement learning (RL), evolutionary algorithms (EA), and gradient-based approaches. Each strategy possesses its unique strengths and limitations. Compared to RL, EA demonstrates certain advantages in training speed, however, it still entails considerable computational costs. As for gradient-based methods, although they excel in training efficiency, due to the adoption of batch training mode, their global search capability is relatively weak, making them prone to getting stuck in local optimal solutions. Therefore, it is highly necessary to seek a search strategy that is both fast and capable of producing excellent results.

In this paper, we innovatively propose a fast neural architecture search method for multi-modal classification via knowledge sharing, termed KS-NAS. Its core lies in leveraging a knowledge-sharing mechanism to optimize the search process and reduce computational resource consumption. Specifically, we first define a dynamically updated knowledge base, which exists in the form of a supernet and encompasses potential representations of all possible network architectures within the search space. Subsequently, the supernet is trained using a gradient-based optimization strategy to initialize the knowledge base. During the subsequent search process, individuals in each generation of the population acquire initialization parameters from the current knowledge base, thereby avoiding the time-consuming process of training from scratch. These individuals are then trained and optimized in the deep evolutionary search process until convergence. Once this generation of individuals converges, parameters of the outstandingly performing individuals are selected and fed back into the knowledge base for updating. In this manner, the quality of the knowledge base continues to improve as the population evolves, ensuring that subsequent individuals can obtain more excellent initialization parameters. By continuously iterating this process, the KS-NAS method achieves dual enhancements in population quality and training speed. The quality of the initial population in each generation becomes increasingly high, resulting in progressively faster training speeds in each subsequent generation.

We conducted experimental validations on multiple multimodal datasets, and the results indicate that the aforementioned virtuous cycle design strategy not only reduces the consumption of computational resources but also successfully alleviates the prevalent inefficiency issue faced by current EA-based NAS methods. This achievement fully demonstrates the exceptional performance of our method in terms of both efficiency and accuracy. Specifically, our contributions are as follows:

• Individuals can obtain initial parameters from a dynamically updated knowledge base, avoiding the need for training from scratch, which accelerates the Neural NAS process. In traditional NAS methods, each candidate architecture requires training from scratch, which is an extremely time-consuming and resource-intensive process. However, in KS-NAS, by introducing a dynamically updated knowledge base, individuals can directly obtain pre-trained, high-quality initialization parameters, significantly reducing the training time for each generation of individuals.
• The knowledge base is dynamically updated by aggregating the parameters of high-quality individuals in the population, gradually improving its quality. As a core component of the KS-NAS method, the quality of the knowledge base directly affects search efficiency and the accuracy of the final results. After each generation of search, outstanding individuals are selected from the population, and their parameters are integrated into the knowledge base to ensure that it continues to improve its quality as the population evolves. This dynamic update mechanism allows the knowledge base to continuously accumulate excellent network architecture features, providing more valuable initialization parameters for subsequent searches.
• As the population evolves, both the initialization parameters of the population and the quality of the knowledge base improve, forming a virtuous cycle that promotes each other. Compared with existing EA-based NAS methods, KS-NAS maintains high accuracy while achieving faster search speeds and reduced computational load, providing new ideas and methods for addressing the computational challenges in the NAS field.

2 RelatedWork

Multi-Modal Fusion: Multi-modal classification integrates information from different modalities or views to enable models to better understand data and make accurate classifications [Liang et al., 2025; Guo et al., 2024]. Among them, multi-modal fusion plays a crucial role. Within the framework of deep neural networks, multi-modal fusion techniques can primarily be categorized into three major types based on the fusion stage: early fusion, intermediate fusion, and late fusion. Early fusion focuses on integrating low-level features from different modalities at the early stage of data processing [Wang et al., 2018; Yu et al., 2018]. Intermediate fusion refers to the integration of features from different modalities at the intermediate levels of a model [Joze et al., 2020; Vielzeuf et al., 2019]. Late fusion emphasizes the fusion of information from different modalities at the decision level [Han et al., 2020]. In practical applications, the choice of fusion strategy depends on the specific application scenario, data characteristics, and the design objectives of the model.

NAS: NAS aims to automatically discover an optimal network structure [Elsken et al., 2019]. Based on different search strategies, NAS can be mainly divided into the following three categories: RL-based, EA-based, and gradientbased method. (1) The RL-based search strategy [Jaafra et al., 2019; Balaprakash et al., 2019] frames the neural architecture search problem within a Markov decision process framework, employing reinforcement learning algorithms to acquire optimal search policies. A notable disadvantage of these strategy is the high computational cost. (2) The EAbased search strategy [Fu et al., 2024; Han et al., 2024] models neural architecture search as an evolutionary optimization problem, where candidate network structures are regarded as individuals, their performance as fitness, and new individuals are generated through genetic operators, with the optimal individuals being preserved. Although EA are generally faster than RL methods, they still face considerable computational costs. (3) Gradient-based methods [Yu et al., 2022; Ye et al., 2022] require the prior establishment of a supernet, updating parameters by computing gradients of the loss function with respect to the network architecture parameters. They generally exhibit high computational efficiency. Compared to EA, gradient-based methods typically possess weaker global search capabilities and are prone to getting trapped in local optima.

Unlike the aforementioned methods, individuals in KSNAS can acquire initial parameters from a dynamic knowledge base that is initially obtained via training a supernet, thereby avoiding training from scratch; The knowledge base is dynamically updated by aggregating high-quality individual parameters from the population, leading to gradual quality improvement. As the population evolves, the initial parameters of the new individuals become better and better, and so does the quality of the knowledge base; the two reinforce each other. Ultimately, this significantly reduces the computational load and addresses the inefficiency issue of existing population-based NAS methods.

3 Methods

In this paper, we propose the KS-NAS for multi-modal classification to find the optimal multi-modal fusion network structure. The main idea of this method is to construct a dynamic knowledge base for knowledge sharing. By retrieving initial parameters from the knowledge base, we avoid training from scratch, addressing the issue of high computational cost in EA-based NAS. Meanwhile, high-quality individual parameters within the population are aggregated through a fair selection process, dynamically updating the knowledge base and gradually improving its quality, which facilitates the search for the optimal architecture. The overall framework of the KS-NAS is shown in Figure 1.

pernetwork, thereby ensuring that the quality of knowledge stored in the library is continuously improved. This process facilitates the dynamic updating of the knowledge base with high-quality information. Furthermore, individuals within the same generation undergo fair competition, as they all share the same starting conditions so that good individuals will not be selected because of different starting points. The knowledge from previous generations serves as a foundation for the subsequent generations, helping to accelerate their training processes and enabling them to find better solutions. Their synergistic relationship contributes to reducing computational costs and enhancing classification performance.

3.3 Optimal Network Search

We accelerate the training of individuals in the population by sharing knowledge through the knowledge base, adopt the fair selection method to ensure the fairness of individual selection and improve the quality of knowledge in the knowledge base.

First, we define and initialize the supernet as the knowledge base. N subnets are sampled from the supernet, with parameter sharing among them. Features are extracted from multi-modal data using a feature extractor, and then subnets are trained using batch training to update the parameters of the supernet, which also serves as the knowledge base. In the subsequent deep evolutionary search process, subnets are regarded as individuals. Individuals of the same generation acquire knowledge from the same knowledge base for parameter initialization, followed by training. After training for a few generations, evaluation is performed, and the parameters of high-quality individuals are selected to update the knowledge base, facilitating the initialization of individuals in the next generation. The key steps of KS-NAS include population initialization, fitness evaluation, selection, offspring generation, and environmental selection.

3.1 Concept Description

In this section, we explain some of the terms in the paper to avoid conceptual confusion.

Population is a collection of several individuals.

Individual is an integer encoding of a multi-modal fusion network that includes multiple views and fusion operators.

Supernet is a network that contains potential representations of all possible network architectures in the search space and is a concrete representation of the knowledge base.

Subnet is sampled from the supernet and represents the decoded form of an individual.

3.2 Construction of Knowledge Base

The KS-NAS utilizes an evolutionary search approach to find the optimal fusion network. During the evolutionary process, we construct a dynamically updatable knowledge base with the purpose of enabling knowledge sharing among individuals in the population. In this paper, the knowledge base is implemented in the form of a supernet, which contains potential representations of all possible network architectures within the search space. Initially, the knowledge base contains no knowledge, and a gradient-based optimization algorithm is used to train the supernet to initialize the knowledge base.

As the evolutionary process continues, individuals in the population retrieve initial parameters from the knowledge base, thus avoiding the need to train from scratch. In each generation, all individuals begin with the same starting point. After training and evaluation, high-quality individuals are fairly selected by comparison with the worst fitness of the previous generation, the network parameters of high-quality individuals are integrated to update the knowledge base. The corresponding layer parameters from all high-quality individuals' networks are aggregated through averaging and subsequently integrated with the homologous layers of the su-

Algorithm 1 Pseudo-code of KS-NAS.

Input: Training data Dtrain, validation data Dvalid, feature extractor E.

Parameter: Population size N, maximum number of generations T .

Output: Last generation population PT .

• 1: Extract Vtrain, Vvalid from Dtrain and Dvalid using the well-trained E;
• 2: KS ← Build the knowledge base by training a supernet with gradient-based method;
• 3: Generate initial population P0;
• 4: Initialize the individual network in P0 with KS;
• 5: F0 ←Train and evaluate individual networks of P0 using Vtrain and Vvalid;
• 6: Pm←Record individual networks parameters of P0;
• 7: t ←0
• 8: while t < T do
• 9: Wst ←Find the worst fitness in Ft;
• 10: KS ← Calculate the mean of the Pm and update knowledge base;
• 11: Select mating pool from Pt by the roulette wheel;
• 12: Qt ← Generate offspring by crossing operator and mutation operator
• 13: for each individual q in Qt do
• 14: q ← Initialize by KS;
• 15: end for
• 16: Train and evaluate individual networks in Qt using Vtrain and Vvalid;
• 17: for each individual q in Qt do
• 18: if the fitness of q >Wst then
• 19: Pmq ← Save the parameters of q;
• 20: end if
• 21: end for
• 22: Pm←Pmq;
• 23: Pt+1 ←Select N individuals fromPt and Qt by environment selection;
• 24: Ft+1 ←Record the fitness of Pt+1;
• 25: t ← t + 1;
• 26: end while
• 27: return PT .

Population initialization: Individuals are specifically implemented in the form of subnets, each of which is composed of input views and fusion operators arranged in a certain fusion order. There is a schematic diagram representing the subnet in Figure 2. N subnets are generated through random sampling to serve as individuals in the initial population. Each individual selects k non-repeating features from the available view features, thereby avoiding local optimal solutions caused by abnormal concentration of feature information in the initial population. The number of fusion operators used in each individual is k - 1, which corresponds to the number of fusion nodes for the selected feature.

Fitness evaluation: Each individual is decoded into a multi-modal fusion network. The detailed decoding process is shown in Subsection 3.4. Each network is initialized with the knowledge base and then trained with the dataset. The adaptive value is evaluated after the training.

Selection: We use roulette wheel selection to determine which individuals to mate with. The specific step is to generate a random number r ∈ [0, 1], and then select the first individual whose cumulative probability exceeds the random number r. Through this approach, individuals with higher fitness have a greater chance of being selected, while also preserving the possibility of individuals with lower fitness being selected, helping to avoid falling into local optimal solutions.

Crossover and mutation: The CSG-NAS method inspires us [Fu et al., 2024] to design an adaptive crossover rate and mutation rate, two parameters that are constantly adjusted over the course of evolution. The crossover rate gradually increases with evolutionary generations, while the mutation rate decreases accordingly. A higher initial mutation rate enhances population diversity and broadens the search space to prevent premature convergence. As evolution progresses, elevating the crossover rate facilitates effective gene recombination for fine-grained local optimization. This mechanism balances global exploration and local exploitation, ensuring both solution quality and convergence efficiency.

Environment selection: After combining parents and children, we adopted a meritocracy selection strategy to select the next generation of parent individuals to ensure that the best individuals reach the next generation and continue to pass on their good genes. The elite selection method effectively avoids the possible information loss in genetic manipulation by preferentially retaining the individuals with the highest fitness. At the same time, this strategy can accelerate the convergence process, reduce unnecessary fluctuations in the evolutionary process, and ensure the stable transmission of excellent genes.

The pseudo code of KS-NAS is shown in Algorithm 1.

3.4 Encoding and Decoding

In the population, each individual is encoded using integers and consists of multiple nodes. Each node is comprised of three digits, where the first two digits represent input features, and the third digit indicates the fusion operation performed on the two input features. The node itself represents the result of the fusion. The fusion operations include addition, multiplication, concatenation, maximize, and average. The relevant definitions can be found in work [Liang et al., 2021]. Before decoding each individual into multi-modal fusion network, it is necessary to align the modal features in a fully connected way to facilitate fusion in the fusion network. Then, the corresponding modal features are fused according to the fusion network generated by decoding. After fusion, the final fusion results are transmitted to the FC and Softmax layers for the final output. The detailed explanation is shown in Figure 2.

4 Experiments

4.1 Datasets

All of our experiments are implemented using Torch 2.4.1 on Ubuntu 16.04.4. It is equipped with 512GB DDR4 RDIMM memory, two 40-core Intel Xeon CPU E5-2698v4 @ 2.20GHz processors, and uses an NVIDIA Tesla P100 GPU. The effectiveness of our method is verified on five popular multi-modal datasets. The following is the introduction of the datasets: (1) ChemBook-10k (CB) [Liang et al., 2021] dataset, which is a chemical structure image recognition dataset for patent search research. The dataset consists of 1 million images of chemical structures belonging to 10,000 categories. (2) The NUS-Wide-128 (NUS) [Tang et al., 2016] dataset contains 43,800 single-label images from 128 categories. In our experiment, we used a subset of this dataset containing 23,438 images from 10 categories, each associated with a label, with at least 1,500 images per category. (3) The MM-IMDB dataset [Arevalo et al., 2017] is a multi-modal dataset collected from the Internet Movie Database, including 25,959 movies and their associated posters, plots, genres, and other metadata, comprising a total of 27 non-mutually exclusive genres. Due to a severe class imbalance, we chose to use only 23 types for the classification task, and the dataset was divided into 15,552 movies for training, 2,608 movies for validation, and 7,799 movies for testing purposes. (4) NTU RGB-D [Shahroudy et al., 2016] is a large-scale multi-modal action recognition dataset with 56,880 samples, divided into 60 categories in total. The training, validation, and test sets included 23,760, 2,519, and 16,558 samples, respectively. (5) EgoGesture [Zhang et al., 2018] is a multi-modal gesture recognition task dataset containing 24,161 gesture samples, with a total of 83 categories. There were 14,416 samples in the dataset for training, 4,768 samples for validation, and 4,977 samples for testing.

4.2 ComparisonMethods

To better validate the effectiveness of KS-NAS, we compared it to a variety of methods including several SOTA algorithms. (1) Single-mode method: Inflated ResNet-50 [Baradel et al., 2018], Co-occurrence [Li et al., 2018], VGG-16+LSTM [Yang and Tian, 2014], C3D+LSTM+RSTTM [Molchanov et al., 2016], I3D [Carreira and Zisserman, 2017], ResNext101 [K¨op¨ukl¨u et al., 2019], Maxout MLP [Goodfellow et al., 2013], VGG Transfer [Simonyan, 2014].

(2) Traditional multi-modal methods and advanced fusion operators: TMC [Han et al., 2022], TMOA [Liu et al., 2022], AWDR [Yang et al., 2019], RAMC [Jiang et al., 2022], Twostream [Simonyan and Zisserman, 2014], GMU [Arevalo et al., 2017], CentralNet [Vielzeuf et al., 2019], MMTM [Joze et al., 2020], MTUT [Gupta et al., 2019], MBL [Kim et al., 2016], MFB [Yu et al., 2018], TFN [Zadeh et al., 2017], LMF [Liu et al., 2018], PTP [Hou et al., 2019].

(3) Multi-modal fusion method based on NAS: EDF [Liang et al., 2021], MFAS [P´erez-R´ua et al., 2019], BM-NAS [Yin et al., 2022], 3D-CDC-NAS2 [Yu et al., 2021], DC-NAS [Liang et al., 2024] and CSG-NAS [Fu et al., 2024].

4.3 Experiments Results and Analysis

For the CB and NUS datasets, we employ a five-fold crossvalidation approach to partition the data into training and testing sets, allowing a more accurate estimation of the model's performance and generalization ability. To ensure a fair comparison with other multi-modal fusion methods, we follow the setup of EDF [Liang et al., 2021], using the same feature extractors for data processing and the same fusion operator for constructing the search space. We compare KS-NAS with several advanced multi-modal fusion operators and methods, among which only EDF, DC-NAS and CSG-NAS are NASbased. The results are shown in Table 1, indicating that KSNAS achieves state-of-the-art classification performance. On the CB dataset, our method outperforms the advanced NASbased multi-modal methods EDF, DC-NAS, and CSG-NAS by 4.75%, 4.69%, and 4.01%, respectively. On the NUS dataset, it surpasses them by 1.97%, 1.44%, and 1.12%, respectively. This suggests that knowledge sharing can indeed improve the performance of multi-modal classification tasks.

For the MM-IMDB dataset, to ensure a fair comparison with other multi-modal fusion methods, we follow the setup of BM-NAS [Yin et al., 2022], using Maxout MLP as the backbone model for the text modality, and VGG Transfer as the backbone model for the RGB image modality. The evaluation metric used is the weighted F1 score, which is a reliable indicator for measuring multi-label classification performance due to the highly imbalanced nature of the dataset, rather than other types of F1 scores. The use of the weighted F1 score is also consistent with previous methods for easy comparison. For the parameters of our architecture, we set the population size to N = 20, the number of iterations to T = 20, the dimension of the fusion vector FD = 256, and the modality features are all reproducible. As shown in Table 2, KS-NAS outperforms existing multi-modal classification methods in terms of the weighted F1 score, surpassing MFAS, BM-NAS, DC-NAS, and CSG-NAS by 4.07%, 3.65%, 2.87%, and 2.45%, respectively.

For the NTU RGB-D dataset, to ensure a fair comparison of methods, we follow the setup of BM-NAS, using the dilated ResNet-50 as the backbone model for the video modality and Co-occurrence as the backbone model for the skeleton modality. This design ensures that all methods in the experiment share the same backbone network. For KSNAS, the experimental settings are a population size of 20, 20 iterations, a fusion dimension of 256, and reusable modality features. As shown in Table 3, our method achieves a cross-subject accuracy of 91.11%. Compared to other methods, KS-NAS performs slightly worse than CSG-NAS but still outperforms most advanced multi-modal methods, surpassing MFAS, BM-NAS, and DC-NAS by 1.61%, 0.63%, and 0.26%, respectively.

For the EgoGesture dataset, to ensure a fair comparison of methods, we follow the setup of BM-NAS, using ResNeXt101 as the backbone model for both the RGB and depth video modalities. This design ensures that all methods in the experiment share the same backbone network. We compare KSNAS with various unimodal and multi-modal methods. The experimental settings for KS-NAS include a population size of 20, 20 iterations, a fusion dimension of 256, and reusable modality features. As shown in Table 4, our method achieves a cross-subject accuracy of 95.27%. Compared to other methods, KS-NAS achieves state-of-the-art classification performance.

In summary, KS-NAS, by sharing knowledge through a knowledge base, has improved model performance to a certain extent and provided new perspectives and implementation strategies for multi-modal classification tasks.

4.4 Search Efficiency Comparison

We analyze KS-NAS from three aspects: model parameters, search efficiency, and classification performance. To better validate the capability of KS-NAS, we compare it with various powerful multi-modal fusion methods, with experimental results shown in Table 5. From the table, it can be observed that although KS-NAS has a larger model size, it achieves superior classification performance while requiring the least search time.

On the CB and NUS datasets, the search efficiency of KSNAS is nearly three and four times faster than the state-of-theart method CSG-NAS, with better performance. On the MMIMDB dataset, KS-NAS outperforms state-of-the-art methods in terms of search efficiency and classification performance, and has a small number of parameters. Although on the NTU RGB-D dataset, our method's performance is slightly lower than CSG-NAS, its search efficiency is twice as fast, and its performance is still better than other advanced multi-modal fusion methods. On the EgoGesture dataset, KSNAS also achieves the highest search efficiency and excellent classification performance. These results demonstrate the superiority of KS-NAS in terms of efficiency.

4.5 Ablation Study

To better analyze KS-NAS, we conducted ablation experiments on the CB and NUS datasets. The results are shown in Tables 6 and 7. They reveal that, when comparing the classification performance of KS-NAS1 and KS-NAS2, the knowledge-sharing (KS) is effective in improving search efficiency and reducing search time. As shown in Figure 3, under the same conditions, with the KS from the knowledge base, the number of training iterations required by individuals gradually decreases as the population evolves, leading to an improvement in search efficiency. However, due to the variability in the quality of stored knowledge, this approach can lead to local optima, resulting in suboptimal classification performance. The standard deviations indicate that KS-NAS2 has lower stability, as the uneven quality of knowledge causes unfair training evaluations for different structures.

Comparing the data between KS-NAS2 and KS-NAS shows that the inclusion of a fair selection (FS) method improves search efficiency, classification results, and stability. This improvement is due to fair selection ensuring the quality of the knowledge in the knowledge base, allowing different structures to be trained to their specified limits in a short time with similar starting points, increasing the chances of finding better structures and enhancing classification performance. Finally, it is found that under the joint action of knowledge sharing and fair selection, the search efficiency is improved by ten times or even greater, and the classification performance is also improved to some extent.

5 Conclusion

In this paper, we propose a knowledge sharing-based neural architecture search (KS-NAS) method to reduce the timeconsuming nature of current NAS-MMC methods during the training process. The KS-NAS method constructs a dynamically updatable knowledge base for knowledge sharing, enabling individuals to obtain initial parameters from the knowledge base and avoiding the need for training from scratch. High-quality individuals, in turn, contribute to enhancing the quality of the knowledge base, creating a mutually beneficial relationship between the two. This method significantly reduces computational overhead and improves the efficiency of existing population-based NAS-MMC methods. Finally, the experimental results on multiple popular multimodal tasks demonstrate that our method achieves stateof-the-art results in terms of classification performance and training efficiency. In the future, we will conduct in-depth research on improving the quality of the knowledge base, such as narrowing the search space based on core structures to control the direction of knowledge base updates and designing metrics to quantify the quality of the knowledge base numerically.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Nos. 62406218, 62306171), Fundamental Research Program of Shanxi Province (No. 202203021222183), Science and Technology Major Project of Shanxi (No. 202201020101006), Open Project Foundation of Intelligent Information Processing Key Laboratory of Shanxi Province (No. CICIP2023005), Taiyuan University of Science and Technology Scientific Research Initial Funding (No. 20222106), Reward funds for outstanding doctor of work in coming to Jin (No. 20232029).