Deep Multi-Dimensional Classification with Pairwise Dimension-Specific Features

1 Introduction

In multi-dimensional classification (MDC), each object is represented by a single instance while associated with multiple class variables. Here, each class variable corresponds to one label space characterizing the rich semantics of objects from some specific dimension. Take landscape paintings classification as an example, each picture can be classified from time dimension (with possible labels morning, afternoon, night, etc.), from whether dimension (with possible labels sunny, rainy, cloudy, etc.), and from scene dimension (with possible labels desert, mountain, grass, etc.). Specifically, the needs of learning fromMDC objects widely exist in diverse real-world applications, including text mining [Lertnattee and Theeramunkong, 2004; Shatkay et al., 2008], computer vision [Song et al., 2018; Lian et al., 2020; Shi et al., 2025], bioinformatics [Borchani et al., 2013; Fernandez Gonzalez et al., 2015], etc.

∗Corresponding author

Formally speaking, given a feature space X = Rd and an output space Y = C1 × C2 × · · · × Cq corresponding to the Cartesian product of q label spaces, each label space Cj = {cj1 , cj2 , . . . , cjKj } includes Kj possible labels (1 ≤ j ≤ q) to characterize the semantics along one dimension. Let D = {(xi, yi)|1 ≤ i ≤ m} be the training set and each sample (xi, yi) corresponds to a d-dimensional feature vector xi = [xi1, xi2, . . . , xid]T ∈ X associated with a q-dimensional label vector yi = [yi1, yi2, . . . , yiq]T ∈ Y. Given an unseen instance x∗, the task of MDC is to learn a mapping function f : X ↦→ Y from the training set D which can return a proper label vector f(x∗).

One popular solution to MDC tasks is to independently deal with each dimension as a traditional multi-class classification problem. Nonetheless, this strategy completely ignores the dependencies among class variables and then the performance of the induced predictive model might degenerate. To tackle this issue, existing MDC approaches are designed to consider class dependencies in either explicit manner with some structures (e.g., directed acyclic graph [Bielza et al., 2011; Gil-Begue et al., 2021], chaining order [Zaragoza et al., 2011] and pairwise interaction [Jia and Zhang, 2020b]) or in implicit manner via manipulating feature space [Jia and Zhang, 2020a; Jia and Zhang, 2022] or label space [Jia and Zhang, 2021b; Tang et al., 2024].

Although these existing approaches have successfully considered class dependencies, they might obtain suboptimal performances since the predictive models for different dimensions are induced based on the same feature space [Jia et al., 2023]. However, different semantics in each dimension might prefer different feature characteristics. Take the aforementioned landscape painting as an example, the level of luminance would be preferred in discriminating labels in time dimension; abrupt color changes are more likely to reveal the labels in scene dimension and the upper part of a picture is supposed to be more related to whether dimension. Moreover, samples belonging to the same class should be similar in the feature space generally, but it is very common that two MDC samples belong to the same class in one dimension while different classes in another dimension.

To consider the specific characteristics contained in different semantic dimensions as well as the dependencies among class variables, we propose a novel MDC approach named PIST (i.e., Pairwise dImension-Specific feaTures) based on deep learning techniques. Specifically, PIST aims to model class dependencies between each pair of class variables via pairwise grouping. Firstly, to construct pairwise dimensional embeddings, a combinatorial encoding procedure is conducted for pairwise class variables via optimizing the intraclass and inter-class covariance. Then an element-wise selection mechanism is used to extract pairwise dimension-specific features, which are considered better capturing the correlation between feature space and heterogeneous label semantics in respective dimension pairs. Finally, joint probabilities predicted by pairwise neural networks are integrated to accomplish the final discrimination collectively. To the best of our knowledge, PIST serves as the first attempt towards learning dimension-specific features as well as considering class dependencies. Comprehensive experiments on eleven benchmark data sets show that PIST performs better than existing well-established MDC approaches.

The rest of this paper is organized as follows. Firstly, Section 2 briefly reviews related works. Then Section 3 presents the proposed PIST approach at length. After that, Section 4 reports the results of empirical studies over a wide range of MDC data sets. Finally, Section 5 concludes this paper.

2 RelatedWork

On one hand, MDC can be regarded as a set of multi-class classification problems, one per dimension. Thus, we can solve the MDC problem by learning an independent multiclass classifier for each dimension, which is known as binary relevance (BR) [Zhang et al., 2018] but ignores all possible class dependencies. To exploit class dependencies among dimensions, on the other hand, one straightforward strategy is to regard each distinct label combination as a new class, which is known as class powerset (CP). However, the combinatorial nature of CP induces high complexity and is also prone to class-imbalance and overfitting problem. According to the strategy of dependency modeling, existing MDC works can be roughly categorized into two categories, including explicit and implicit dependency modeling methods.

The MDC methods of explicit category attempt to model class dependencies with some explicit structures. Chainingbased classifiers improve BR by learning a chain of multi-class classifiers, where subsequent classifiers on the chain will augment predictions of preceding one as features [Zaragoza et al., 2011; Read et al., 2014b]. Multidimensional Bayesian classifiers construct directed acyclic graph over class variable to explicitly consider the class dependencies [Bielza et al., 2011; Gil-Begue et al., 2021]. Generally, the dependencies among many class variables are hard to model due to limited samples in training set. SEEM [Jia and Zhang, 2020b] and MDKNN [Jia and Zhang, 2021a] suggest that pairwise dependencies can be modeled more reliably than modeling the dependencies among all class variables.

The MDC methods of implicit category attempt to transform the original MDC problem into a new one without some explicit dependency modeling mechanism in the transformation procedure. gMML [Ma and Chen, 2018] transforms the original categorical output space into a binary one and then induce the predictive model based on metric learning.

SLEM [Jia and Zhang, 2021b] encodes the original class vectors into real-valued ones and decodes the predicted class vectors over the outputs of learned multi-output regression model. To extract more powerful features, KRAM [Jia and Zhang, 2020a] manipulates the feature space via utilizing kNN information to enrich the original feature space.

LEFA [Wang et al., 2020] is the first MDC approach that utilizes deep learning techniques. It learns an augmented feature vector for each instance via assuming that the representations of features and labels should be aligned in some latent space. ADVAE-FLOW [Zhang et al., 2022] encodes both feature and class variables to probabilistic latent spaces by normalizing flows, in which the one-hot representation for the label vectors w.r.t. different dimensions are directly stacked. DSOC [Saleh and Li, 2023] is formed of multiple neural networks and a hypercube classifier, where the former are responsible for feature selection and the latter aims to accommodate the model for rare sample classification.

However, all these works only aim to consider class dependencies but cannot consider the specific characteristics contained in different semantic dimensions. In the next section, we will present the technical details of the proposed PIST approach, which considers not only class dependencies but also pairwise dimension-specific characteristics.

3 The PIST Approach

As shown in Figure 1, PIST includes two key modules: pairwise dimension encoding and dimension-specific feature extraction. Briefly, a weighted sum-pooling is conducted to obtain pairwise dimension embeddings in the first module and the outputs will further guide the dimension-specific feature extraction in the second module. The final classification is enabled by the returned probabilities of softmax regression.

3.1 Pairwise Dimension Encoding

To consider pairwise interactions, PIST considers each pair of dimensions as an entirety. Without loss of generality, we will carry out the following discussions in the case of C1 and C2. To extract dimension-specific features for this dimension pair, PIST learns a corresponding pairwise dimension embedding with a weighted sum-pooling as follows:

where fa1a2 is the number of samples labeled by c1a1 and c2a2 in the training set and la1a2 ∈ Rt is a latent label embedding vector related to c1a1 and c2a2 . Here, t is a hyper-parameter to be set (cf. Section 4.3 for further discussions).

For each component la1a2 in Eq.(1), a natural cluster assumption is that la11, la12, . . . , la1K2 are close to each other and far from l˜a11, l˜a12, . . . , l˜a1K2 where a1 ∈ {1, 2, . . . ,K1} and ˜a1 ∈ {1, 2, . . . ,K1} \ {a1}. For this purpose, good embeddings la1a2 should minimize the intra-class covariance and maximize inter-class covariance, which can be implemented via minimizing the following objective L(12) le-part1 :

where ¯ la1 = 1 K2 ∑K2 a2=1 la1a2 is the intra-class mean and ¯ l = 1 K1K2 ∑K1 a1=1 ∑K2 a2=1 la1a2 is the global mean.

It is worth noting that the above discussions on similarity of latent label embeddings have another 'dual' form with an exchange of subscript a1 and a2. Then another objective L(12) le-part2 is as follows:

where ¯ l′a2 = 1 K1 ∑K1 a1=1 la1a2 is the 'dual' intra-class mean. By combining the two above objectives together, the final label embedding loss L(12) le w.r.t. C1 and C2 can be defined as follows:

3.2 Dimension-Specific Feature Extraction

For each example x, numerous early works [Yeh et al., 2017; Wang et al., 2016; Zhang et al., 2023] have proved that it is significant to exploit powerful feature embeddings in latent spaces. Thus we firstly encode x into a latent space based on neural networks which is denoted by ϕ(x) ∈ Rd′ . To extract pairwise dimension-specific feature, decode l(12) into a feature importance vector with an attention network:

where θ(12) , bl ∈ Rd′ and Wl ∈ Rd′×t . σ is the ReLU activation function. Here, note that Wl and bl are shared parameters for all pairwise dimensions.

PIST further assumes that ϕ(x) could be transformed via an element-wise selection mechanism. Then for dimensions C1 and C2, the latent feature embedding is transformed into ϕ(x) ⊙ θ(12) , where ⊙ is the Hadamard product. With a fully-connected network used, the final pairwise dimensionspecific feature is obtained:

where s(12) , bs ∈ Rd′ , Ws ∈ Rd′×d′ .

In addition, it is possible that one dimension is irrelevant to others. PIST also seeks to acquire single dimension-specific features by a similar procedure. Take the j-th dimension as an example (1 ≤ j ≤ q):

Here, we denote j by jj for notation consistency with the aforementioned pairwise case.

3.3 Classification

For classification, we simply obtain probabilities of all K1K2 class combinations w.r.t the first two dimensions with a softmax regression as follows:

where o(12) , b(12) o ∈ RK1K2 and W(12) o ∈ RK1K2×d′ . Define an injective function ψ(·, ·) : {1, 2, . . . ,K1} × {1, 2, . . . , K2} → {1, 2, . . . , K1K2} and further assume that ψ(a1, a2) = w. The predicted probability of any instance x is as follows:

where the w-th element ˆ p(12) w in ˆ p(12) corresponds to:

Here, o(12) a denotes the a-th element in vector o(12) a . It is easy to know that ˆ p(12) w indicates the probability that x is labeled by c1a1 and c2a2 w.r.t. C1 and C2, respectively.

Similar derivation can apply to the case of a single dimension. Take the j-th dimension as an example (1 ≤ j ≤ q):

where o(jj) , b(jj) o ∈ RKj and W(jj) o ∈ RKj×d′ . The corresponding predicted probability of any instance x is as follows:

where the aj-th element ˆ p(jj) a in ˆ p(jj) corresponds to:

After traversing all dimension pairs, we can obtain (q2) + q predicted probabilities {ˆ p(rs) |1 ≤ r ≤ s ≤ q}, the final confidence score ρ(r) ar for the ar-th label in the r-th dimension is determined as follows (ar ∈ {1, 2, . . . , Kr}, 1 ≤ r ≤ q):

It is not hard to verify that ∑Kr ar=1 ρ(r) ar = q holds. To render ρ(r) ar probabilistic and facilitate cross-entropy loss, we further normalize it with softmax operation:

Based on Qrar , assuming ground-truth label of x in the r-th dimension is crγ , the cross-entropy loss w.r.t. the r-th dimension is defined as follows (1 ≤ r ≤ q):

where �π� returns 1 if π holds and 0 otherwise. The final loss corresponds to the sum of the average of the dimension-wise cross-entropy loss L(r) ce in Eq.(18) as well as the average of pairwise label embedding loss L(rs) le in Eq.(4):

Given an unseen instance x∗, its predicted label ˆ y∗j w.r.t. the j-th dimension is determined as follows (1 ≤ j ≤ q):

The final predicted vector can be obtained after traversing all dimensions, i.e., ˆ y = [ˆ y∗1, ˆ y∗2, . . . , ˆ y∗q]T.

4 Experiments

4.1 Experimental Setting

Data Sets

In this paper, we use eleven real-world MDC data sets for experimental studies. Table 1 summarizes basic characteristics, including the number of examples (#Exam.), the number of dimensions (#Dim.), the number of labels in each dimension (#Labels/Dim.) and the number of features (#Feat.).

Evaluation Metrics

In this paper, three commonly used metrics for performance evaluation are adopted, i.e. hamming score (HS), exact match (EM) and sub-exact match (SEM) [Read et al., 2014a; Zhu et al., 2016]. Given the test set S = {(xi, yi) | 1 ≤ i ≤ p} and the MDC model f to be evaluated, the definitions of these three evaluation metrics are given as follows:

1. Hamming Score:

1. Exact Match:

3. Sub-Exact Match:

Here, r(i) = ∑qj=1 �yij = ˆ yij� denotes the number of dimensions which are predicted correctly, yij and ˆ yij denote the ground-truth and predicted label w.r.t. the j-th dimension for the i-th test sample. Ten-fold cross validation are conducted for all data sets where the mean metric value as well as the standard derivation are recorded for comparison.

Comparing Approaches

We compare PIST against eight state-of-the-art MDC approaches with parameter configurations suggested in respective literatures:

Table 2: Experimental results (mean±std.) of each MDC approach. In addition, ·/◦ indicates whether PIST is significantly superior/inferior to other compared approaches on each data set with pairwise t-test at 0.05 significance level.

• BR: Learn an independent multi-class classifier for each dimension one by one.
• CP: Learn a multi-class classifier via treating each distinct label combination as a new label.
• ECC [Zaragoza et al., 2011]: Ensemble of several multiclass classifier chains with random dimension orders. Predicted results generated by preceding classifiers are taken as augmented inputs of the subsequent classifier.
• gMML [Ma and Chen, 2018]: Mapping the output space in MDC into a binary one via one-vs-rest strategy. The resulted problem is solved by learning regression models

based on metric learning.

• KRAM [Jia and Zhang, 2020a]: Count the number of instances in the k nearest neighbors of each sample which is associated with exactly each label respectively w.r.t. each dimension. These counted numbers are concatenated to form augmented features.
• LEFA [Wang et al., 2020]: Introduce a cross correlation aware network to learn low-dimensional latent label embeddings which are considered close to latent feature embeddings. Aligned label embeddings are used to augment the original feature space. Multi-class algorithms

are also used for subsequent classification.

• MDKNN [Jia and Zhang, 2021a]: Obtain kNN counting statistics as KRAM and consider class dependencies for each pair of label spaces. Predictions are determined by the best learned classifier which achieve the highest accuracy in the k nearest neighbors.
• SEEM [Jia and Zhang, 2020b]: Learn pairwise classifiers in the first level and stack corresponding predicted outputs according to the accuracy in the k nearest neighbors to generate second-level data sets for subsequent multiclass models.

For comparing approaches which necessitate a multi-class algorithms, LIBSVM [Chang and Lin, 2011] is used to implement the base classifier as suggested in literatures. While PIST is based on neural networks, to make fair comparison and eliminate the impact exerted by difference of base classifiers, we further investigate 5 adjusted approaches including BR, KRAM, LEFA, MDKNN and SEEM by replacing the multi-class classifier with neural networks as PIST. In the context below, these methods with changed base classifiers are denoted by the original name plus a subscript δ. Detailed implementation will be elaborated in the next section.

Implementation Details

For our proposed method, feature embeddings are generated by a fully-connected layer and ReLU activation. Label embeddings {laras |1 ≤ r ≤ s ≤ q, 1 ≤ ar ≤ Kr, 1 ≤ as ≤ Ks} are initialized by standard normal distribution. It is worth noting that we adopt a dropout-like strategy for all label embeddings used for weighted sum-pooling, i.e. randomly drop 80% of them to alleviate overfitting. The latent dimensions of label embeddings t, feature embeddings d′ and all hidden layers are empirically set as 32, 512 and 512, respectively. All activation functions are fixed as ReLU followed by a dropout layer [Srivastava et al., 2014] with dropping probability of 0.5. For network optimization, SGD with a batch size of 512 and momentum of 0.9 is employed. We set the learning rate as 0.1 and the weight decay as 10-4 .

For comparing approaches, all recommended parameters in their literatures are employed. Given that in PIST, the subsequent networks adopted on transformed features {ϕ(x) ⊙ θrs |(1 ≤ r ≤ s ≤ q)} are actually equivalent to a multi-layer perceptron with one hidden layer (Eq.(6) and Eq.(10)), thus for changed base classifiers, we replace all LIBSVM implemented classifiers with the exact same multi-layer perceptron for all comparing approaches.

4.2 Experimental Results

The detailed experimental results are reported in Table 2. Due to the space limitation, the detailed experimental results of MDC approaches with replaced base classifiers are deferred into the supplementary materials. Moreover, pairwise t-test [Demˇsar, 2006] at 0.05 significance level is conducted to show whether PIST achieves significantly superior/inferior performance against other comparing approaches on each data set. Accordingly, the resulting win/tie/loss counts are summarized in Table 3 and Table 4.

According to the reported experimental results, observations can be made as follows:

• Evaluated by three metrics, PIST respectively significantly outperforms the 13 comparing approaches (8 original approaches plus 5 approaches with neural network based classifiers) in 84.6%, 74.6% and 70.0% cases across all the 130 configurations (11 data sets × 13 comparing approaches).
• PIST achieves greater advantage on large-scale data sets than on small-scale data sets probably because deep learning technique is more suitable for sufficient data. Relatively poor performance on Scm20d might be attributed to the large number of dimensions which is a challenging circumstance for pairwise strategy.
• Substituting neural network based classifiers for SVM presents overall degeneration of performance, which might be caused by the inconsistency between the transformed data sets generated by original MDC approaches and neural networks.

To summarize, PIST achieves highly competitive performance against other well-established MDC approaches, which validates the effectiveness of our proposed pairwise dimension-specific feature learning approach.

4.3 Further Analysis

Ablation Studies

In this section, we conduct ablation studies on all the eleven MDC benchmark data sets. The four variants are denoted as PAIR, RAND, STACK, and MONO.

• PAIR. Use original features to induce pairwise classifiers instead of learning dimension-specific features.
• RAND. Initialize pairwise dimension embeddings l(ij) by standard normal distribution without combinatorial strategy in Eq.(1) and thus remove the embedding loss (i.e. the second term in Eq.(19)).
• STACK. To reduce the number of neural network based classifiers, stack the dimension-specific features for all dimension pairs. Specifically, for the j-th (1 ≤ j ≤ q) dimension, the stacked dimension-specific feature is:

Here, s(1j) , . . . , s(jq) are obtained in Eq.(6) and Eq.(9). Then for the obtained q dimension-specific features {s(j) |1 ≤ j ≤ q}, q fully-connected layers are respectively used to output probabilities w.r.t. corresponding dimensions.

• MONO. Get rid of all pairwise parts, i.e., only employ Eq.(7)∼Eq.(9), Eq.(13)∼Eq.(15).

Wilcoxon signed-ranks test [Demˇsar, 2006] at significance level α = 0.05 is conducted to analyze whether PIST performs statistically better than variant models. Table 5 summarizes the p-value statistics on each evaluation metric. Compared with these four variant models, we observe that PIST

achieves statistically superior performance against them in terms of each metric, which validates effectiveness of PIST in the following aspects:

• It is beneficial to use pairwise dimension-specific features than original ones.
• Our proposed combinatorial encoding method with embedding loss generates better label embeddings for capturing the correlation between features and their corresponding dimensions compared to random initialization of label embeddings.
• Combining probabilistic predictions w.r.t. pairwise dimensions can effectively leverage class dependencies, surpassing the strategy of stacking dimension-specific features and obtaining prediction in each respective dimension directly.
• Single dimension-specific features are insufficient to model class dependencies and our proposed pairwise dependencies modeling is one of the most essential approaches for designing MDC algorithms.

Parameter Sensitivity

Figure 2 shows how the performance of PIST fluctuates with different values of t, i.e. the dimension of latent label embeddings as mentioned in Eq.(1). It is shown that PIST achieves relatively stable performance when the value of t changed in the range of {25 , 26 , 27 , 28 , 29 }. In this paper, the value of t is set to 25 in light of lower complexity which can be used as the default parameter setting.

5 Conclusion

The main contributions of this paper are two-fold. (1) We propose to consider the specific characteristics contained in different semantic dimensions for MDC which is as important as modeling class dependencies. (2) We proposed a novel MDC approach named PIST which learns pairwise dimension-specific features for MDC to consider both the specific characteristics in each dimension and the dependencies among different dimensions. Experiments clearly validate the effectiveness of the proposed PIST approach.

Acknowledgements

The authors wish to thank the anonymous reviewers for their helpful comments and suggestions. This work was supported by the National Science Foundation of China (62225602, 62306131), and the Big Data Computing Center of Southeast University.