Advocacy Learning: Learning through Competition and Class-Conditional Representations

1 Introduction

In recent years, researchers have proposed a large number of modifications to the standard supervised learning setting with the goal of improving performance [Parascandolo et al., 2018; Vaswani et al., 2017]. These modifications focus on training different parts of the network (i.e., subnetworks) to cooperate. However, in several real-world settings, such as in economics and law, agents who compete are critical for identifying good solutions. While recent work in adversarial networks investigates the use of competition for training models, the final model evaluated (i.e., the generator) is cooperative [Goodfellow et al., 2014]. In contrast, we investigate a training scheme in which subnetworks compete during training and evaluation. In our model, subnetworks compete to provide evidence in the form of class-conditional attention maps. Here, we use the term 'attention map' to refer to a filter that indicates parts of the input that are useful for accurate classification, similar to the idea of saliency [Itti et al., 1998]. An example of a standard network architecture with attention is given in Figure 1a. We hypothesize that class-conditional attention maps could offer advantages over standard attention maps by emphasizing portions of the input indicative of their class.

Our proposed approach consists of two main subnetworks: a single Judge and multiple Advocates (see Figure 1b). Each Advocate produces an attention map that advocates for a particular class. A decision is reached by the Judge, which weighs the arguments produced by the Advocates. For this approach to work well, there must be a balance among the Advocates (so that each Advocate can influence the Judge), and the Judge must be able to effectively use the given evidence (so as not to be deceived by the advocates). We achieve this balance via advocacy learning, which trains the components jointly, but according to multiple objectives. These different objectives are key to striking the right balance between providing strong but factual evidence. We also explore a variant, honest advocacy learning, where the Advocates are not trained to deceptively compete with one another, but still provide class-conditional attention maps. In a series of experiments, we compare advocacy learning to several baselines in which the entire network is trained according to the same standard objective. Across several image datasets, we observe a small but consistent improvement in classification accuracy by using class-conditional attention maps.

2 Methods

We propose a novel approach to optimizing networks for supervised classification that encourages class-conditional representations of evidence in the form of attention maps. We hypothesize that, depending on how they are learned, classconditional attention maps could offer advantages over standard attention maps by encouraging competition among components of the network. At a high level, the Judge learns to solve the classification problem given some evidence, while each Advocate supplies that evidence by arguing in support of their class. This setup disentangles evidence supporting each class, encouraging strong class-conditional representations. Advocacy learning consists of both a specific architecture (i.e., Advocate and Judge subnetworks) and a specific method for training, both are described below.

2.1 Problem Setting

We consider the task of solving a multi-class classification problem in a supervised learning setting with element-wise attention. We assume access to a labeled training set consisting of labeled examples {x, y}, where x ∈ Rd (where d may be a product d1 × d2, such as in an image) and y ∈ {1, . . . ,N}, where N is the number of classes. We refer to the one-hot label distribution entailed by y as y, so y[y] = 1 and y[j] = 0 for all j ̸= y. We use square brackets for indexing into a vector. We focus on deep learning methods due to their suitability for representation learning. We indicate the parameters of deep models as θ and subscript them according to the particular subnetwork being referenced. The parameters of the Judge are θJ and the parameters for each Advocate i are θi where i ∈ {1, . . . ,N}. Our proposed approach aims to solve the multi-class classification problem through a novel training scheme, designed to produce classspecific evidence.

2.2 Network Architecture

As mentioned above, our proposed approach is composed of two sets of modules: multiple Advocates and a single Judge. A high-level overview of our architecture, which we call an Advocacy Net, is given in Figure 1b. Here, for context, we describe a generic framework, providing specific implementation details in a later section.

Advocate Subnetwork

This subnetwork consists of N Advocates Advi , where i corresponds to the index of the class the Advocate represents. Given the input x, each advocate generates an argument in the form of an attention map Advi (x; θi) → ai ∈ [0, 1]d . The Advocate modules produce an attention map with dimensionality equal to the input using a convolutional encoderdecoder, as is standard for producing pixel-level output in images [Badrinarayanan et al., 2017]. Note that for complex input, such as medical images, other fully convolutional architectures such as U-Nets may be more appropriate [Ronneberger et al., 2015]. Based on these attention maps, each Advocate presents an argument ei (or evidence) to the Judge in the form of an element-wise product between attention maps and the input, ei = ai ⊙ x.

Each Advocate is trained to emphasize aspects of the input indicative of the Advocate's class. This differs from a supervised attention map, which focuses on aspects of the input indicative of the inputs underlying class. In our implementation, Advocates share some underlying evidence in the form of a shared encoder. This allows the Advocates to share useful representational abstractions.

Judge Subnetwork

The Judge J takes as input the combined evidence E = [e1, . . . , eN] ∈ RN×d , and outputs a probability distribution over classes J(E; θJ) → ˆy. We make specific class predictions using argmax(ˆy). The architecture of the Judge is flexible; the only limitation is that the input size must be proportional to the total number of classes. In our implementation, the Judge module is a convolutional network with fully connected output layers.

Though certain constraints on the architecture of the network are necessary, the interplay among the modules and how they are trained are key to the advocacy learning framework. Trained end-to-end with the objective of minimizing training loss, there would be no difference between the proposed architecture and a network with multiple attention channels. In the next section, we describe the key differences in how we train the Advocates vs. the Judge.

2.3 Training Algorithm

The complete advocacy learning algorithm is presented in Algorithm 1. We learn the parameters of the Judge subnetwork by minimizing the cross-entropy loss: CE(ˆy, y) = -log ˆy[y], as is standard for classification. The Advocates are trained according to a different objective. In particular, Advocate i is trained by minimizing the advocate crossentropy loss: CEA(ˆy, i) = -log ˆy[i]. Under this objective, each Advocate is trained to represent samples from all classes

as its own. We also consider a variant, called honest advocacy

Algorithm 1: Advocacy Learning Algorithm

Input : Labeled training data D= {xk, yk}Sk=1 where S is the number of samples, xk ∈ Rd , and yk ∈ {1, . . .N} Output: Trained Network A = (J, Adv1 , . . . AdvN ) 1 Initialize parameters for Judge θJ and Advocates θ1, . . . , θN; 2 while training do /* Showing single sample for simplicity */ 3 Draw example (x, y) ∼D; 4 for i ∈ {1, . . .N} do 5 ai ← Advi (x; θi); 6 ei = ai ⊙ x; 7 end 8 E← [e1, . . . , eN]; 9 ˆy ← J(E; θJ); 10 LJ = -log(ˆy[y]) ; // Cross-Entropy Loss 11 θJ ← θJ - η ▽θJ LJ; 12 for i ∈ {1, . . .N} do 13 if not honest or i = y then /* Honest Advocates update on true examples */ 14 LAdvi = -log(ˆy[i]); 15 θi ← θi - η ▽θi LAdvi ; 16 end 17 end 18 end 19 return A = (J,Adv1 , . . . ,AdvN )

learning, in which the Advocates are not trained to deceive, but aim to minimize:

We optimize the parameters of the Judge and Advocates by making a prediction and then interleaving steps of gradient descent, updating the Judge and each Advocate individually according to their respective loss functions. At each step, we freeze the parameters of all other subnetworks.

3 Baselines &Experimental Setup

We evaluate our proposed advocacy learning approach across a variety of tasks and compare against a series of different baselines. In this section, we explain each baseline and provide implementation details.

3.1 Model and Baselines

On all datasets, we compare (honest) advocacy learning against two baselines that incorporate attention:

• Attention Net: this baseline modifies the Advocacy Net architecture by removing all but one attention module. The one remaining module is trained using a standard end-to-end optimization approach minimizing cross entropy loss. This allows us to compare advocacy learning against a similar model using standard supervision.
• Multi-Attention Net: two differences exist between Attention Nets and Advocacy Nets: the optimization procedure and the architecture. To tease apart these two aspects, we include a comparison against a model with an identical architecture, but trained using a standard endto-end loss.

3.2 Implementation Details

We implement our models using PyTorch [Adam et al., 2017]. Our specific model architecture (number of layers, filters, etc.) is available via our public code release 1 .

In our experiments, we optimize the network weights using Adam [Kingma and Ba, 2014] with a learning rate of 10-4 , and use Dropout and batch normalization to prevent overfitting. We split off 10% of our training data to use as a validation set for early stopping. We cease training when validation loss fails to improve over 10 epochs. Model performance is reported on the canonical test splits for each dataset. We regularize the attention maps by adding a penalty proportional to the L1-norm of the map to encourage sparsity consistent with common notions of attention. Parameters were initialized using the default PyTorch method.

4 Results and Discussion

We begin by examining the performance of advocacy learning across three image datasets, analyzing when and how advocacy learning impacts performance. We then present performance on a significant real-world medical dataset and modified datasets designed to highlight the effects of competition and deception in learning. We conclude by providing some intuition for why advocacy learning works.

4.1 Advocacy Learning onMulti-Class Balanced Image Data

We begin by examining the performance of our Advocacy Net variants and baselines on two publicly available image classification datasets: MNIST and Fashion-MNIST [Xiao et al., 2017]. As described above, the parameters of the Advocate modules are not optimized to improve overall classification performance, and as a result their inclusion could lead to a reduction in performance (by deceiving the Judge). However, we find that this is not the case (Table 1). On these two datasets, advocacy learning does as well as or outperforms all baselines. The improvement is most pronounced in FashionMNIST, perhaps due to the denser images or the baseline performance allowing room for improvement. Moreover, this difference is not solely attributable to the class-conditional nature of the attention-maps; across both datasets the Advocacy Nets outperform the Honest Advocacy Nets (test accuracy 99.42 vs. 99.32 in MNIST and 91.62 vs. 90.81 in FMNIST). This suggests that deception, in addition to competition, can help produce useful attention maps.

We also examined performance on a more challenging image classification problem, CIFAR-10. When using a similar architecture and training procedure as in the above datasets, we found the Advocacy Net improved over other approaches

1 https://github.com/igfox/advocacy-learning

(in particular, test accuracy 83.47 vs. 79.73 for the MultiAttention Net), though both approaches performed poorly relative to state-of-the-art. Thus, we examined replacing the Judge network with a ResNet-110, using a fixed-epoch training scheme with learning rate decay (similar to a popular open source implementation 2 ). This increased the performance of the Multi-Attention Net and drastically lowered the Advocacy Net performance (test accuracy 30.54 vs. 92.01), suggesting that under our current training scheme Advocacy Learning is unstable when the Judge is high-capacity relative to the Advocates. Notably, Honest Advocacy Learning continues to perform well, beating the Multi-Attention Net (test accuracy 92.68 vs. 92.01).

4.2 Impact of Class Conditional Attention

Results in Table 1 demonstrate that class-conditional attention can improve upon supervised attention maps. Honest Advocacy Nets perform similarly to Multi-Attention Nets on MNIST and slightly better on FMNIST (99.32 vs. 99.33 and 90.81 vs. 90.11). These architectures differ only in that the attention maps in the Honest Advocacy Net are classconditional. The competition introduced by the Advocacy Net further improves performance, outperforming the MultiAttention Net on both datasets.

To further compare advocacy learning with the end-to-end supervised baselines, we plot the averaged difference between the confusion matrices of the Multi-Attention Nets and Advocacy Nets (Figure 2). Overall, we observe that Advocacy Nets result in improvements for a subset of class pairs (e.g., classes 4 and 9), but leave the majority of predictions unchanged. On MNIST (Figure 2(a)), we find a few examples where advocacy learning lowers performance. In particular, the Advocacy Net is more likely to misclassify 8s as 9s; though the reverse error rate (9s as 8s) does not increase. This is likely due to the asymmetric morphological relationship among the digits: using per-pixel attentions in [0, 1], an 8 can be obscured to look like a 9, but the converse is less likely. The Advocacy Net appears to help emphasize curves in the input (reducing the instances with 9 classified as 4 or 7 classified as 9). On Fashion-MNIST (Figure 2(b)) we observe that certain class pairs (pullovers or coats vs. shirts) are markedly improved, while most others are unaffected. This suggests that advocacy learning improves performance by distinguishing among classes with similar morphology; though this analysis is confounded by the fact that it tends to be those class pairs that have the greatest potential room for improvement.

2 https://github.com/bearpaw/pytorch-classification

Qualitative examples of attention maps from the Honest Advocates and Multi-Attention Net are given in Figure 3. We found Honest Advocacy Nets gave denser (and thus more interpretable) attention maps than Advocacy Nets. We observe that both Advocacy Nets and Multi-Attention Nets generate a variety of attention maps with checkering characteristic of deconvolutional layers [Odena et al., 2016]. An interesting example of class-conditional behavior is shown by the advocate for class 1. This Advocate, representing the "pants" class, emphasizes the sides of the shirt, similar to pants legs.

4.3 Balancing the Advocates and Judge

Since the Advocates are encouraged to deceive the Judge, a natural question arises - How does the relative capacity of these components impact performance? To answer this question, we consider a variation of our Advocacy Net, in which we can vary the capacity of different pieces. We replace the convolutional layers in the Advocate encoder and Judge with some number of convolutional residual blocks, as in [He et al., 2016]. By changing the number of blocks, we can increase/decrease the capacity of the Advocate or Judge. Architectural details of this modification can be found in our code. We varied the number of residual blocks from 1-5 and 1-3 in the Judge and Advocates respectively. We observed that the Advocacy Nets achieved the highest classification accuracy when Judge capacity was high and Advocate capacity was low (best accuracy 99.46). We performed an identical architecture search with the Multi-Attention Net; there was no capacity setting that beat the best results attained by the Advocacy Net (best 99.34% vs. 99.46%). This suggests that it is important for the Judge to have more capacity than the Advocates; though our experiments on CIFAR suggest that a balance must be struck. While a high-capacity Judge is better able to use evidence provided by the Advocates, it may train slowly, and thus be more susceptible to deceptive Advocates.

To better understand the impact of deception in advocacy learning, we took a fully trained network and examined the effect of freezing the Judge while continuing to update the Advocates for both advocate and Honest Advocacy Nets. We found that training without the Judge did not affect network performance in the Honest Advocate Net, but decreased performance in the advocate net by 85%. Thus, adaptations by the Judge play a crucial role in maintaining advocate net accuracy.

Up to this point, we have considered only Advocacy Nets in which all Advocates share an encoder. Such an architecture could encourage implicit sharing of information, possibly tempering the negative effects of deception. To test this hypothesis, we evaluated an Advocacy Net without a shared encoder on MNIST and FMNIST. On both datasets (averaged across 5 runs), the Advocacy Net without a shared encoder achieved lower performance both in absolute terms and relative to an Honest Advocacy Net without a shared encoder (98.29 vs. 99.05 for MNIST, 86.47 vs. 89.29 for FMIST). This suggests the shared encoder is an important way for Advocates to balance performance.

4.4 Generalization to other Settings

The results presented so far all involve multi-class image datasets with balanced classes. To explore how these assumptions affect the performance of advocacy learning, we applied advocacy learning to a large electronic health record (EHR) dataset, MIMIC III [Johnson et al., 2016]. This dataset, a publicly available repository of EHR data, has become an important benchmark in the machine learning for health community [Harutyunyan et al., 2017], and is helping to drive advances in precision health [Desautels et al., 2016;

Maslove et al., 2017]. We used the clinical time-series subset of the database for mortality prediction, as in [Harutyunyan et al., 2017]. We also considered variants of MNIST that break the multi-class and balanced assumptions. These additional experiments test the generalizability of our findings to i) different data types (time series as opposed to images), ii) imbalanced classes, and iii) binary labels. Our results are presented in Table 2.

We report our results on MIMIC in terms of the area under the precision recall curve (AUPR) and the receiver operating characteristic curve (AUROC), since the task is binary with considerable class imbalance in the test set (13.23% positive). Notably, in this task advocacy learning performs slightly worse than all baselines. However, honest advocacy learning provides a small benefit relative to the baselines in terms of AUPR.

This reversal of the results from Table 1 is interesting, and helps illuminate cases where advocacy learning may or may not work. There are many differences between MIMIC and MNIST/FMNIST that could explain why advocacy learning applied to MIMIC fails. First, this task involves classifying patients based on a feature vector of vital sign measurements. Our attention mask allows for input values to be reduced to some proportion between 0 and 1. This type of attention can allow for pictures of shirts to look like pants, or an 8 to look like a 9, but it may not allow for vitals from sick patients to look like vitals from healthy patients (or vice-versa). Other major differences, besides the data type, include class imbalance and the number of classes. As these are qualities that can be imbued in other datasets, we further examine them by modifying MNIST to create new datasets we call Imbalanced

MNIST and Binary MNIST.

In Imbalanced MNIST, we subsampled the training set, introducing class imbalance. After subsampling, the least represented class, 0, had 600 training samples, and each successive class had 600 additional samples. The test set remained unchanged. We found that class imbalance in the training set lowered the performance of all models by 0.1-0.3% relative to those same models performance with balanced training data; the Advocacy Net was more strongly affected than the Honest Advocacy Net. However, both models resulted in similar accuracy (advocacy learning 99.17 ± 0.14 vs. honest advocacy learning 99.17 ± 0.06).

Binary MNIST contains only two classes: 4 and 9. The per-class number of examples in the training and test set were unchanged. We found that the switch to a binary formulation reduced absolute performance for the Advocacy Net by 0.47% relative to the performance of the full network evaluated only on 4s and 9s (99.19 vs. 98.72), whereas the Honest Advocacy Net improved binary performance by 0.32%. This decrease suggests that, in practice, the competition among many Advocates helps the Judge achieve good performance in the presence of deception.

In all datasets, the class-conditional attention provided by honest advocacy learning did not hurt, and in the presence of imbalanced data helped. This suggests the value of classconditional representations, with or without deception.

5 RelatedWork

The fact that advocacy learning, which encourages deceitful subnetworks, works at all, let alone better than the baselines in some tasks, may be surprising. Several recent works, however, have found competition useful for learning [Goodfellow et al., 2014; Silver et al., 2017; Sabour et al., 2017]. Specifically, adversarial relationships, or situations where subnetworks compete with one another, have recently garnered interest [Goodfellow et al., 2014; Ghosh et al., 2018]. For example, Ghosh et al. examine a multi-generator setting, where each generator is encouraged to capture distinct portions of the class distribution. This resembles how advocates captures relevant evidence for their corresponding class. However, while the relationship between an Advocate and the Judge is not entirely cooperative (the Advocate may argue for an untrue class), neither is it entirely adversarial (the Advocate may argue for a true class). While these systems used competition among networks during training, competition within a network has been used as well. A winner-take-all competitive framework was found to lead to superior semi-supervised image classification performance [Makhzani and Frey, 2015], and the dynamic routing used in Capsule Networks can been seen as type of competition [Sabour et al., 2017].

Beyond this work in competition, advocacy learning is related to both i) input transformation and ii) debate agents. First, others have considered transformations of the input in the context of improving classification. Parascandolo et al. proposed the use of a mixture-of-experts model to learn inverse data transforms to improve performance. The competition among experts to provide quality transforms of the input resemble the competition among Advocates to convince the Judge. Our work differs in that: 1) our Advocate updates are unsupervised, 2) the goal of the Advocates is not to improve performance, and 3) Advocates are assigned classes. Hong et al. examined the use of class-conditional attention for improving semantic segmentation. Their attention maps are similar to the argument maps generated by Advocacy Nets, but are trained with an end-to-end objective. Similarly, Capsule Nets [Sabour et al., 2017] use class-specific capsules at the output layer to define a class-conditional parse of the input. This can be viewed as a bottom-up analog to the classspecific attention maps produced by Advocates, but it trained end-to-end unlike advocacy learning.

Second, work in AI debate sets up a similar task to our own, training agents to 'debate' in order to convince a Judge about the class associated with an input [Irving et al., 2018]. The authors use Monte-Carlo tree search to simulate a debate with the goal of identifying a series of pixels to convince a pretrained Judge classifier of an input example's class. While conceptually similar to advocacy learning, our work differs in motivation and methodology. In addition to considering a different learning framework: neural networks, vs. Monte-Carlo tree search, there are two key differences in problem formulation: 1) our work does not use a pre-trained Judge, and 2) our work involves Advocates, which are assigned classes, not debaters, which choose classes.

6 Conclusion

We have presented a novel approach to attention-based supervised classification: advocacy learning. Our approach divides a network into two subnetworks: i) a set of Advocates trained to provide arguments supporting their corresponding class, and ii) a Judge that uses these arguments to predict the true class. Over a series of experiments on three publicly available datasets, we showed that class-conditional attention can improve performance relative to standard attention, and that in some circumstances competitive training can further improve performance.

The use of a deep network for the Judge has important implications for the interpretability of the derived attention maps. If the Judge is a high-capacity nonlinear network, then the evidence it may find convincing will, by default, be uninterpretable. However, the flexibility of the proposed architecture means that work on training interpretable networks or interpreting trained networks applies [Ribeiro et al., 2016]. Future work could consider using these techniques to create interpretable Advocacy Nets. Extensions may also consider improving the balance between cooperation and competition by controlling the ratio of honest and deceptive updates, or the ratio of class-specific updates. Currently, the proposed architecture is limited by the one-to-one relationship between the number of classes and number of advocates, which makes training on datasets like ImageNet infeasible. Future work could examine methods to remove this linear relationship, such as training advocates that work across class hierarchies. While there are many avenues for future improvements, the experiments explored in this work suggest that competition and class-conditional representations can in some cases be in used to improve the utility of attention in classification tasks.