ASCENT-ViT: Attention-based Scale-aware Concept Learning Framework for Enhanced Alignment in Vision Transformers

1 Introduction

With the surge of advanced deep learning (DL) methods, deep neural networks (DNNs), especially Transformer-based networks [Vaswani et al., 2017], have been widely deployed in human communities for the benefit of our life [Dargan et al., 2020; Shorten et al., 2021]. Research in developing highperforming, generalizable, and efficient architectures has led to experts classifying a family of fundamental architectures as 'Foundation Models' [Bommasani et al., 2021]. In computer vision, the most widely utilized foundation model, Vision Transformers (ViT), shows great scalability and impressive performance in various downstream tasks, which promotes its use in real-world applications like autonomous driving [Ando et al., 2023]. However, with the development and application of such advanced DNNs, there is a growing concern about their lack of interpretability leading to hesitation in applying the technology in high-stakes areas such as medical diagnosis, facial recognition, finance, etc. Given the current situation, several studies on the interpretability of DNNs have emerged [Li et al., 2022].

Current research on DNN interpretability can be broadly categorized into two classes - dubbed backward and forward alignment [Gabriel, 2020]. Backward alignment is post-hoc interpretation, which explains already trained models. Specifically, some post-hoc methods assign relative importance scores to different features considered [Ribeiro et al., 2016; Sundararajan et al., 2017], while some other approaches will rank the input training samples according to their importance to prediction [Koh and Liang, 2017; Ghorbani and Zou, 2019]. On the other hand, forward alignment methods try to design and train intrinsically interpretable models. One such approach attempts to incorporate human-understandable concepts into model architecture and training processes. Concepts can be considered as shared abstract entities across multiple instances, which provide a general understanding of the task processed by the model. Many studies have explored how to integrate the model's decisionmaking process with task-relevant concepts to achieve transparent and interpretable model predictions [Koh et al., 2020; Pedapati et al., 2020; Jeyakumar et al., 2020; Heskes et al., 2020; Chen et al., 2019]. Formally, given a set of concept representations C associated with an input sample x, the task prediction is y = g(C), where g is any interpretable function used to aggregate concepts. Multiple works such as [Alvarez Melis and Jaakkola, 2018] and [Agarwal et al., 2021] have postulated using a weighted linear sum of concepts as g. Other works such as [Koh et al., 2020] have postulated using a shallow feed-forward network as mapping from concepts to prediction.

With the success of attention mechanism [Vaswani et al., 2017] in designing large-scale networks for language [Devlin et al., 2018] and vision applications [Dosovitskiy et al., 2020], multiple diverse use-cases for attention have been proposed. A particular use case of attention in concept-based explainability is utilizing the attention mechanism in modeling the concept aggregation function g as first proposed in [Rigotti et al., 2021]. Note that concepts are independent entities shared among different training samples, hence attention can be reliably used to assign "relevance" scores to the concepts relative to a prediction.

ViTs [Dosovitskiy et al., 2020] demonstrate significant improvement over traditional models like CNNs for largescale vision applications. ViTs borrow their architecture from transformer-based language approaches [Vaswani et al., 2017; Devlin et al., 2018]. A ViT first segments an image into equal patches, which are projected into an embedding space with positional information. The representation is then passed through stacked transformer blocks which is used in downstream tasks. ViTs have demonstrated better performance than CNNs, especially when pre-trained with largescale data. However, due to the very architecture of ViTs being borrowed from language paradigm, they lack visionspecific inductive biases like CNNs. CNNs have been specifically designed for image modalities, and capture explicit inductive biases such as - scale invariance and transformation equivariance. As an example, ViTs are demonstrated not to be scale-insensitive [Xu et al., 2021], a known strength of CNNs. Researchers have attempted to improve ViTs by incorporating CNN-like inductive biases in their architectures with great success [Liu et al., 2021; Graham et al., 2021].

However, concept-based explainability approaches are still designed to be model-agnostic [Koh et al., 2020] and plugand-play [Rigotti et al., 2021] to fit all possible use cases. As incorporating various inductive biases in the model architecture provides significant performance improvement, a detailed design decision should be made for concept-based explainability modules as well. For instance, explainability modules for ViTs can benefit from scale awareness in addition to patch awareness during concept learning in ViTs. As a consequence, we design a novel framework for ViTs that incorporates desirable properties (inductive biases) of ViTs - patch awareness and CNNs - scale invariance. Specifically, in this paper, we propose Attention-based Scale-aware Concept Learning Framework for Enhanced Alignment in Vision Transformers (ASCENT-ViT), a concept-based explainability module for ViTs to effectively align humanannotated concepts with model representations. Unlike model-agnostic plug-and-play explainability, ASCENT-ViT effectively composes inductive biases of CNNs and ViTs - scale invariance and image-patch relationship awareness respectively into scale and patch-aware representations. More precisely, ASCENT-ViT consists of 3 distinct modules - (i) Multi-scale Encoding (MSE) Module, which models concepts at various scales, (ii) Deformable Multi-Scale Fusion (DMSF) Module, which composes multi-scale concepts with patch embeddings, and (iii) Concept-Representation Alignment Module (CRAM), which aligns concepts with the learned model representations. To summarize, our contributions are:

• We propose ASCENT-ViT, a concept-based explainability method that effectively composes inductive biases from CNNs and ViTs to perform better conceptrepresentation alignment. We are the first work to effec-
• tively integrate these inductive biases in concept-based explainability modules.
• We demonstrate through quantitative experiments that ASCENT-ViT improves both prediction accuracy and concept learning as compared to model-agnostic attention-based concept explainability modules across 7 different vision-transformer architectures.
• We demonstrate through diverse real-world visual examples that ASCENT-ViT captures concept annotations missed by model-agnostic approaches and is robust to transformations and perturbations.

2 RelatedWork

Vision Transformers (ViTs) and Inductive biases. Multiple approaches have improved the original ViT [Dosovitskiy et al., 2020] performance [Atito et al., 2021], efficiency [Chen et al., 2022] and explainability [Xu et al., 2023]. Even though ViTs offer better downstream task performance as compared to CNN-centric approaches [Tuli et al., 2021; Raghu et al., 2021], the consensus among researchers remains that ViTs have much less stringent inductive biases and some approaches improve them [Xu et al., 2021]. To alleviate this problem, namely - lack of scale invariance and transform equivariance, DETR [Zhu et al., 2020] proposes an attention mechanism to incorporate scale features while SWIN [Liu et al., 2021] uses a shifted window approach in ViTs with great success. A few approaches have improved ViT architectures with scale-awareness [Lin et al., 2023; Guan et al., 2024].

Concept-based Explainability and Attention. Conceptbased explanations explain models using humanunderstandable concepts [Koh et al., 2020; Sinha et al., 2023; Sinha et al., 2024a]. Concepts are high-level attributes shared across different instances [Chen et al., 2019]. Explorations uncover the decision-making of neural networks via concepts, including learning latent concept scores [Alvarez Melis and Jaakkola, 2018], finding concept prototypes [Li et al., 2018; Chen et al., 2019], and activations [Sinha et al., 2024b].

Comparison with related approaches. Recent approaches incorporate an external Attention matrix to align model representations with human-understandable concepts. Two related works to our method ASCENT-ViT are - Concept Bottleneck Models (CBMs) [Koh et al., 2020], CT [Rigotti et al., 2021], and BotCL [Wang et al., 2023]. Both CBMs and BotCL utilize an intermediate layer to perform alignment between concepts and representations but neither utilize spatial concept annotation maps - CBMs discretize the concepts to binary labels while BotCL is completely unsupervised and utilizes slot attention to learn representative concepts. For completeness, we use CBM as one of the baselines to compare our method's performance. The closest work to ASCENT-ViT is CT [Rigotti et al., 2021], which utilizes a model-agnostic plug-andplay module to align concepts with representations. However, there are significant differences between our approach and CT. Firstly, their module is model-agnostic and does not account for the inductive biases. Secondly, their method does not incorporate the effect of scale - making it fragile to minor transformations. Finally, their method does not provide any analysis on its effectiveness over ViT scale and architecture - a very important determinant of the extent of misalignment.

3 Methodology

In this section, we introduce our proposed Attention-based Scale-aware Concept Learning Framework for Enhanced Alignment in Vision Transformers (ASCENT-ViT). ASCENTViT is composed of three independent modules, namely - the Multi-scale encoding (MSE) module which extracts multi-scale features for enhanced scale awareness, the Deformable Multi-Scale Fusion (DMSF) Module which incorporates image scales using Deformable Attention, and the Concept-Representation Alignment Module (CRAM) which aligns scale and patch aware features with concept annotations. The ASCENT-ViT can be utilized as a classification head on top of standard ViT backbones for improved conceptbased explanations. The overall schematic representation of the ASCENT-ViT is shown in Figure 1.

Problem Setting: A concept-based model consists of networks f and g. For a given training sample {(xi, yi,Ci)}, where xi ∈ RD denotes the i-th training sample, yi ∈ R is the target classification label for sample xi, and Ci ∈ RT is a vector of T human annotated concepts, the network f learns a mapping from image space to the concept space, i.e., f : RD → RT , while the network g maps concepts to the prediction space, i.e., g : RT → R. The training procedure entails supervising the concept space with human-annotated concepts and promoting the alignment of concepts and model representations. In our method, the concepts Ci are the concept maps with pixel-level annotations.

3.1 Multi-scale EncodingModule

The Multi-scale encoding Module consists of a Convolutional Neural Network (CNN) F which can be thought of as S stacked convolution blocks (with implicit activation and pooling operations) fi, such that F = fS ◦ fS-1 ◦ fS-2...., ◦f1. CNNs embody inductive biases in the form of convolution

Deformable Multi-Scale Fusion (DMSF) Module

Figure 2: (a) Detailed viewof theMSEmodulewhich extractsmultiscale features {ci}S1 and concatenates them together into a vector c. (b) Detailed view of the proposed DMSF module utilizing Deformable Attention operation to combine multi-scale features c with patch embeddings from ViT (zq) to output scale and patch-aware representations z.

operations which capture the local semantics of images at various scales, i.e., each block computes feature maps representing various scales. As a consequence, for a given input image x ∈ RD , we extract S multiscale features from various blocks of F. Mathematically, each feature for a particular scale can be computed as

Subsequently, the combined multiscale features are concatenated by the MSE module to output a combined and spatially aware multi-scale feature c. Mathematically, the scale-aware feature c is given as,

Note that the feature c is sometimes called the feature pyramid [Lin et al., 2017; Chen et al., 2022] because it intuitively represents multiple scales of the same image, each scale progressively smaller than the previous scale. The architecture of the MSE module is presented in Fig. 2 (a).

3.2 Deformable Multi-Scale FusionModule

The DMSF module consists of two distinct steps. The first step computes an attention operation using deformed (scaled) outputs c from the MSE module with the Multi-Scale Deformable Attention (MSDA) operation [Zhu et al., 2020] as discussed below. The second step composes a list of patch outputs zq from a ViT backbone with multi-scale features computed from the MSDA operation using a combination of a learnable parameter (I) and a tunable hyperparameter (ψ). Multi-scale Deformable Attention (MSDA) Operation. As discussed before, the fundamental problem in the standard attention mechanism used in ViTs is the lack of scale and spatial awareness. To incorporate multi-scale feature maps, [Zhu et al., 2020] proposed MSDA. The MSDA operation is characterized by a set of reference points (depicted by coordinates pq and their offsets from a reference point ∆p) shared among features corresponding to multiple scales of the same image's feature maps which localizes semantics across scales. Mathematically, the MSDA operation for a set of featuremaps of various scales {ci}S1 over a query-sample zq, the MSDA(zq, pq, c) operation can be represented as,

where M is the number of attention heads (indexed by m), S is the number of scales (indexed by i), K is the number of sampled reference points (indexed by k). W projects Value vector to multiple heads while W′ is the inverse projection. A denotes the attention matrix, and ∆pmiqk represents offset corresponding to sampled reference points pq, for each head and scale. Correspondingly, each element in A is referenced as head (m) and scale (i), while q and k represent the query and key indices, implying the number of reference points sampled is S ∗ K. The function ϕ scales up/down the reference points to corresponding scales as ci. Intuitively, the module can be understood as utilizing multi-scale features as key/value vectors that transform a patch-aware vector (zq) into a patch and scale-aware vector (z).

Adaptive Patch Composition. Note that the output from the ViT backbone is represented as zq (Figure 1), which is treated as the query vector for the MSDA operation. Even though the MSDA operation introduces scale awareness in the outputs of the ViT backbone, a tradeoff between patch-specific semantics and scale awareness is desirable due to the wide gulf in the inductive biases in them. To alleviate this, we introduce a learnable vector I which controls the contribution of scale features, and a scalar hyperparameter ψ which weighs the effect of the MSDA operation on the final patch vector. Mathematically, the final output of the DMSF module is as follows:

where norm represents the layer norm, zq represents the input from the ViT backbone, pq represents the reference points while c = {ci}S1 represents the multi-scale features computed by the MSE module. Note that adaptive composition is sensitive to the initialization of the I vector and the tunable hyperparameter ψ. An overview of the DMSF module is shown in Fig. 2 (b).

3.3 Concept-Representation Alignment Module

The Concept-Representation Alignment Module (CRAM) is inspired by the Attention mechanism proposed in [Vaswani et al., 2017] wherein, an attention matrix Aatt computation is conditioned on three vectors of the same dimensions - Query (Q), Key (K), and Value (V). Mathematically, the attention operation is as follows:

The attention matrix attends to each value in the Query and Key vectors with a composite score. CRAM treats the Query vector as the patch representation z from the DMSF module and the Key Vector as Human-annotated Concepts C. As patches and concepts are continuous and one-hot respectively, projection matrices convert them to continuous vectors. Mathematically, the attention matrix for patch embeddings (z) and concepts (C) associated to an input sample x is calculated as follows:

where the matrices Pq and Pk are learnable projection matrices transforming patches and concepts to Query and Key vectors respectively, while dim represents the dimension of the patch and concept embeddings. Intuitively, each entry in the matrix AQK encapsulates the contribution of a concept on a patch and is a measure of alignment. The final prediction is then calculated by multiplying the attention matrix with concept projections (V = CPk) followed by a final learnable projection matrix (Pv) to map to logits. Mathematically each logit ˆ y in the prediction ˆ y can be calculated as an average of concept contributions on each patch.

where |z| stands for the dimension of z. In practice, two different attention matrices (ASpatial and AGlobal - the QK subscript is omitted for clarity) are calculated for different subsets of concepts - spatial and global. The spatial concepts can be localized, while the global concepts are more semantic and cannot be localized. The final output is then the sum of logits computed from both spatial and global attention matrices. Mathematically the sum can be written as:

where Pv1 and Pv2 are different versions of Pv associated with ASpatial and AGlobal, respectively. The detailed description of CRAM is shown in Figure 3.

3.4 End-to-end Training of ViTwith ASCENT-ViT

The overall schematic pipeline of our approach is depicted in Figure 1. As MSE, DMSF and CRAM modules are standalone modules that can be appended on top of any standard ViT backbone. To sum up, the inference step for an input image x utilizing ASCENT-ViT computes the patch embeddings zq through the following procedure:

Training Objective. The final training objective includes the task prediction objective utilizing a classification loss. For datasets with explicit localization of concepts in the image (e.g. CUB), the attention matrix ASpatial can be regularized by human-annotated localizations. Mathematically, the training objective can be given as:

where L represents any classification loss, H denotes humanannotated localization points for the concepts, λ is a tunable hyperparameter and the subscript F is the Frobenius norm.

4 Experiments and Results

4.1 Dataset Description

(1) CUB200 [Wah et al., 2011]: The Caltech-UCSD Birds200-2011 dataset consists of 11,788 photos of 200 different classes of birds annotated with concepts representing physical traits of birds like wing color, beak size, etc. (2) AWA2 [Xian et al., 2018]: Animals with Attributes-2 consists of 37,322 images of a combined 50 animal classes with 85 binary concepts like number of legs, presence of tail, etc. We utilize Segment Anything Model (v2) 1 to annotate 10 selected concepts. (3) KITS [Heller et al., 2023]: The Kidney Tumor Segmentation (KITS) challenge dataset consists of images of kidneys where the task is segmentation of renal cysts and tumors. We augment each image with the concepts from the metadata, namely - kidney, tumor, and cysts. We sample 10000 images with an 80:20 train/test split. (4) PascalAPY [Farhadi et al., 2009]: We utilize Pascal Object Recognition Dataset as processed and described in [Rigotti et al., 2021]. After processing, it consists of 14350 images with 20 attributes each. (5) C-MNIST [Sinha et al., 2023]: Concept-MNIST entails augmenting MNIST dataset with two "spatial" concepts in the form of curved and straight lines. We consider two tasks - even/odd and digit classification.

4.2 Baselines and EvaluationMetrics

Comparision Baselines. As discussed in Related Work, our method is the first to utilize inductive biases in attentionbased concept explainability. The closest comparison to our method is CBM [Koh et al., 2020] (CBMs do not utilize concept attention maps, only global concepts) and [Rigotti et al., 2021] which only utilizes an inferior attention-only architecture similar to only a CRAM module. We denote [Rigotti et al., 2021] as "Only CRAM". We compare our approach on a variety of ViT baselines - with ViT-Base, ViT-Large [Dosovitskiy et al., 2020], SWIN [Liu et al., 2021] for all datasets.

1 https://github.com/facebookresearch/sam2

ViT-Base and ViT-Large are chosen to demonstrate the effect of scale awareness, while SWIN is chosen to demonstrate the effect of inductive bias in the backbone. In addition, we also compare the CUB dataset on DeIT [Touvron et al., 2021], ViT-Base (Dino) [Caron et al., 2021]. SWIN incorporates inductive biases from CNNs. Dino is a robustly trained ViT while DeIT is a distilled version, further capturing the diversity in training procedures.

Evaluation Metrics. The evaluation metric for task prediction is the accuracy in the five datasets. For datasets with spatial concept annotations, namely CUB, AWA2, and KITS, we utilize the Pixel True Positive Rate (Px. TPR) which measures the percentage of correctly identified concept annotations. Due to the extremely precise and sensitive setting of KITS dataset, we measure the total pixel accuracy which encompasses both accurately identified concept and nonconcept pixels. For datasets with no spatial concept annotations (only global) - namely C-MNIST and Pascal APY, we utilize the 0-1 error (misclassification) for concepts, as only binary concepts are considered. Additionally, for CUB200, as localization information for each concept is present, the concept errors are directly evaluated with the Frobenius norm of the difference between ground-truth annotations and predicted concept attention matrices.

4.3 Implementation Details

ViT Backbone. We set the patch size to correspond to 16x16 pixels. Each image is resized to (224,224), and hence a patch sequence is (224/16, 224/16) = (14,14). Following [Rigotti et al., 2021], we append the <CLS>token to the patch sequence after positional embeddings encapsulating global semantics. This results in an input sequence of length of 196+1 = 197 tokens. The internal embedding size (dim) of ViT is 1024.

MSE Module. We utilize three scales in the Multiscale Encoding Module, i.e., S = 3. We utilize scales 1/8, 1/16, and 1/32 corresponding to scale-aware vector c composed of sizes: c1 = 1024, c2 = 196, and c3 = 16, making the size of c = 1029. The first convolution block consists of three convolution layers with batch norm followed by Max pooling. The following convolution blocks consist of a single convolution layer of kernel size=3 and stride=2 with batch norm.

DMSF Module. We utilize Multiscale Deformable Attention [Zhu et al., 2020]. We used 16 attention heads and four reference points, along with layer norms for each key, query, and value vector. We strip <CLS>token before passing through the DMSF module and append it to the output. The initialization value of I is set as 0.01. The value of ψ is tunable and set as 1 for CUB, 2 for Concept-MNIST and Pascal aPY, and 0.5 for AWA2 and KITS.

CRAM Module. We utilize the same embedding dimensions, i.e., dim = 1024 for Key, Query and Value vectors. The number of attention heads is set as 2 for both global and spatial concept attention matrices.

Training Details. We train each dataset and backbone for 50 epochs with early stopping. The maximum learning rate is set at 5e-5 with a linear warmup for the first 10 epochs followed by Cosine decay. The batch size is 16 for each dataset with mixed precision (16-bit default) optimization.

4.4 Quantitative Results

Task Prediction Performance

Table 1 reports the task prediction performance across all five datasets with ViT backbones using CBM (Row-1), augmented with only CRAM (Rows 2-4) and ASCENT-ViT (Rows 5-7). Rows 2-4 list the prediction performance of backbones augmented with only the CRAM module on - ViTBase, ViT-Large, and SWIN. Rows 5-7 list the performance on the same architectures using ASCENT-ViT. ASCENT-ViT outperforms CBMs and only CRAM on all datasets. Interestingly, on SWIN backbones the performance of ASCENT-ViT and 'only CRAM' is on par - implying that SWIN captures scale inductive biases better than ViTs but performs worse than ViT-Large (a larger model). These results demonstrate that both size and inductive biases of ViTs are important factors in improving prediction and concept performances. Our module is effective in introducing scale awareness in standard ViTs and improving their performance.

Concept Annotation Performance

As concept performance metrics are different for datasets with spatial concept annotations and global annotations, we discuss the results individually as reported in Table 1.

True Positive Rate (TPR). For CUB200 and AWA2, we observe that pixel-wise TPR improves by ∼4% on CUB200 and ∼5% implying that ASCENT-ViT captures pixels of interest much better than only CRAM. In Fig 6, we demonstrate the effect on random test images where ASCENT-ViT captures concept annotations more effectively than only CRAM.

Pixel Accuracy. For KITS, we report total pixel accuracy which includes both concept and background annotations. Note that due to the sensitive nature of KITS, it is important to correctly identify both relevant (cysts) and irrelevant concepts (kidney surface). We observe ASCENT-ViT outperforms only CRAM by ∼1.5% over all architectures on KITS. Misclassification Errors. For datasets with only global concepts and no spatial annotations, we report the misclassification error. We observe ASCENT-ViT outperforms only CRAM on both C-MNIST and Pascal aPY.

4.5 Ablation Studies

Hyperparameter Ablations

We demonstrate the effect of the most important hyperparameters in the DMSF module in Figure 4. For all experiments, a ViT-Base model is utilized and trained on the CUB dataset.

Table 2: Task prediction performance ablating on proposed modules of ASCENT-ViT over a single run on tasks. NOTE: ASCENT-ViT implies Backbone w/ MSE+DMSF+CRAM.

Module Ablations

Next, we progressively demonstrate each submodule's effect in Table 2. Line-1 lists the performance of ViT-base with no concept information. Next, Line-2 lists the performance only using the CRAM module. Line-3 lists the performance with both the MSE and CRAM modules. As we observe, adding scale-aware features indeed improves performance. The last line (Line-5) lists performance when the DMSF module performs feature fusion - performing the best.

ViT Backbone Ablations

Table 3 lists the prediction performance of DeIT-Base, SWIN 1x1, and ViT-Base (Dino), variants on CUB200 averaged over three seeds. We observe that ASCENT-ViT outperforms CRAM on models with either a missing Positional or CNNlike inductive bias (DeIT, SWIN, ViT-Base (Dino)). This observation is expected as ASCENT-ViT provides explainability which combines both Positional and CNN-like scaleinvariant inductive biases. Note that on SWIN, which contains pre-encoded CNN-like inductive biases, ASCENT-ViT is on par with CRAM. This observation reaffirms the motivation of our approach - incorporating multiple inductive biases in the explainability module provides better explanations.

Overhead of Explanations. We also provide the overhead of the additional modules as a percentage of model parameters. In most cases, the models augmented with CRAM consist of ≃2% additional parameters than just the backbone implying a negligible increase in the training times.

4.6 Discussion

Figure 6 shows two examples of correctly classified images from CUB200 and one from KITS. The first row (Ground Truth) represents the ground-truth human annotations of the concepts. The second and third rows represent the explanations produced using models trained using only CRAM and ASCENT-ViT on the ViT-Base backbone, respectively. We observe that ASCENT-ViT can better capture regions 'missed' by only CRAM - the brown edge of wings in the first sample and beak shape in the second as compared to ground truth. All the pixels of the cysts in the kidney image are better captured by ASCENT-ViT as compared to only CRAM which misses relevant pixels in the middle of the cyst.

Robustness to transformations. We also report the qualitative performance of ASCENT-ViT in small transformations in Figure 7 - rotation, random cropping, and zoom.

5 Conclusion

In this paper, we propose ASCENT-ViT, a concept-based explainability framework that aligns concepts with representations. ASCENT-ViT captures concepts across multiple scales using a Multi-scale Encoding Module and composes effective multiscale concept-patch relationships through a Multi-scale Feature Interaction Module. Finally, composite representations are aligned with concepts using a ConceptRepresentation Alignment Module. Quantitative and qualitative results demonstrate superior performance over existing methods on multiple ViT backbones. We hope our work helps in designing robust concept-based explainability modules for large-scale DNNs.

Acknowledgements

This work is supported in part by the US National Science Foundation under grants 2217071, 2213700, 2106913,

2008208. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.