BiFSMN: Binary Neural Network for Keyword Spotting

1 Introduction

With the advent of deep neural networks that process speech, great success has been achieved on speech tasks, such as the Deep Feedforward Sequential Memory Networks (DFSMN) [Zhang et al., 2018]. And the deep neural networks for keyword spotting (KWS) are becoming more widely studied for real-world applications on the Internet of Things devices [Chen et al., 2014; Zhang et al., 2015; Ren et al., 2020], which enables users to activate devices by speaking keywords or specific phrases, while keeping the device sleeping when inactive. However, such networks are usually deployed on edge devices with limited computation and power while running on-call to wait for any possible speech, which makes the resources for KWS always seriously constrained. To address the challenge, novel algorithms, such as cFSMN [Chen et al., 2018], DC-CNN [Zhang et al., 2017], and BC-ResNet [Kim et al., 2021], have been proposed for efficient deep learning speedup and memory footprint reduction, they still rely on expensive floating-point operations.

∗ equal contribution † corresponding author

With the most aggressive bit-width, model binarization [Rastegari et al., 2016; Qin et al., 2022] emerges as one of the most promising quantization approaches to compress networks for better computational and storage-usage efficiency. Binary neural networks leverage (1) compact binarized parameters that take limited storage and (2) highly efficient bitwise operations far less costly than their floatingpoint counterparts. However, although the model binarization community has progressed, binarizing the network for KWS

plication of 1-bit parameters, the representation space of the binarized network is extremely limited and hard to optimize. The direct usage of existing binarization methods severely damages the accuracy of binarized models. Second, existing architectures for KWS have fixed model scales and topologies, which cannot adaptively balance the resource budgets at runtime. Moreover, existing deployment frameworks are far from reaching the theoretical upper limit of acceleration for the binarized network when implemented on real-world hardware.

This paper presents a Binarized Feedforward Sequential Memory Network (BiFSMN), which emerges as a successful practice of binary network for KWS application (see the overview in Figure 1), with accurate prediction, lightweight computation, and efficient deployment. BiFSMN is built based on the binarization of D-FSMN, which is a pure feedforward structure. First, we construct a High-frequency Enhancement Distillation (HED) scheme for binarizationaware training, which emphasizes high-frequency information of the full-precision teacher's representation via wavelet transform that is more crucial for the optimization of the binarized network. Second, to enable adaptive accuracyefficiency trade-offs at runtime, we propose a Thinnable Binarization Architecture (TBA) to build various depths (such as [1.0, 0.5, 0.25]×), which further liberates the acceleration potential of the binarized network from the topology perspective to satisfy the different resource constraints. Moreover, we also provide an efficient Fast Bitwise Computation Kernel (FBCK) for BiFSMN on ARMv8 devices. On real-world hardware, the implementation of the BiFSMN using FBCK enjoys impressively faster inference than that using the existing binarization frameworks.

Our BiFSMN is the first binary neural network specialized for KWS. Extensive experiments on Google Speech Commands V1 and V2 datasets (12, 20, and 35 classification tasks) show that our BiFSMN completely outperforms existing binarization methods and is even almost accurate on par with the full-precision counterpart, e.g., BiFSMN just drops within 3% on Speech Commands V1-12. Besides, we highlight that the efficient implementation makes our BiFSMN easy deployment and fast inference in real-world devices: in actual evaluation on edge ARM devices, BiFSMN can achieve up to 22.3× speedup and 15.5× storage-saving compared with the full-precision D-FSMN.

2 RelatedWork

2.1 Network Binarization

Recently, various binarization methods for neural networks have emerged to compress and accelerate networks. The existing binarization methods are designed to obtain accurate binarized networks by minimizing the quantization error [Rastegari et al., 2016], improving loss function [Ding et al., 2019], etc. And from the architectures prospective, despite the most popular CNNs [Liu et al., 2018], transformerbased and MLP-based networks are also studied for binarization [Qin et al., 2021]. The practical use of binarization on real devices relies on deployment support. There are binarization frameworks with different target platforms (e.g., CPUs, GPUs, and FPGAs) and applications, such as daBNN [Zhang et al., 2019] and Bort [Shang et al., 2021].

2.2 Deep Learning for Keyword Spotting

Due to their learning potential and superior performance, deep neural networks for KWS have become widely studied. One classic model is the recurrent neural network (RNN), which enables to capture of the context in a sequence of data [Tian et al., 2021]. CNN-based (BC-ResNet [Kim et al., 2021]) and transform-based (Audiomer) models for KWS are also proposed for better performance and cheaper energy cost. Feedforward Sequential Memory Networks (FSMN) [Zhang et al., 2015] mitigates the vanishing gradient problem of RNNs, and is also efficient in computation in convergence. Improvements on the original design of FSMN are varied, such as compact FSMN (cFSMN) [Chen et al., 2018], Deep-FSMN (D-FSMN), and pyramidal FSMN (pFSMN) [Yang et al., 2018].

3 BiFSMN

In this section, we present the BiFSMN for KWS application. We first build a basic binarization framework and then introduce our techniques, including High-frequency Enhancement Distillation (HED), Thinnable Binarization Architecture (TBA), and Fast Bitwise Computation Kernel (FBCK).

3.1 Basic Binarization Framework

Here we give an introduction to the process of obtaining the basic binarization framework. As one of the most classic and widely used models for speech tasks, D-FSMN [Zhang et al., 2018] is considered as a binarization-friendly architecture for the following reasons: (1) The D-FSMN is a pure feedforward structure with the backbone built up by stacking FIRlike memory blocks. (2) D-FSMN introduces skip connections between memory blocks in adjacent layers to allow the information to flow directly to the next layer, which is also proved important for accurate binarized networks [Liu et al., 2018]. Therefore, we consider the construction of the binarized D-FSMN as the basic binarization framework.

We first introduce the basic formulations of binarization. In the binarized network, both weights and activations are com-

˜pℓt =

+

(1)

where C is the cost function for the minibatch, x denotes the

element in floating-point parameters. The scaling factor α is

highpass

pressed to 1-bit using the sign function in the forward propa-

gation, and the STE [Courbariaux et al., 2015] is applied to

clip the gradient in the backward propagation:

sign(x) =

-1 otherwise , ∂C

∂ sign(x) if |x| ≤ 1

αw = 1n w⊤sign(w) = 1n ∥w∥l1 ,

(2)

we make w ≈ αwBw to reduce the quantization error, where

Bw is the 1-bit weight binarized by sign function.

Then, we introduce the binarized network architecture.

When constructing a binarized D-FSMN, we binarize both

the floating-point weight and activation for linear and con-

volutional layers. Given the input ℓ-th hidden states Hℓ =

[hℓ1 , hℓ2 , ..., hℓT ], and each hℓt ; t ∈ [1, T ] denotes a fixed-size

representation of the long surrounding context at time in-

highpass lowpass

stance t. The formulation of the ℓ-th binarized memory block

takes the following form: (a) original

N ℓ

1∑

i=0

N ℓ

2∑

) + H ( ˜pℓ-1

t ) +pℓt .

Here, pℓt = αV (BVℓ ⊗ Bhℓt ) +bℓ denotes the linear output of the binarized linear projection layer. ⊗ denotes the inner product with bitwise operations XNOR and Bitcount. ˜pℓt denotes the output of the memory block. H(·) denotes the skip connection (identity mapping) within the memory block. N ℓ 1 and N ℓ 2 denotes the look-back and lookahead orders and s1 and s2 are the stride for look-back and lookahead filters respectively. The input of the units for the next hidden layer Hℓ+1 = [hℓ+1 1 , hℓ+1 2 , · · · , hℓ+1 T ] is calculated as follow:

where f(·) = BN · Nonlinear(·) denotes the composition of batch normalization and nonlinear functions (PReLU in the binarized network [Martinez et al., 2020]).

3.2 High-frequency Enhancement Distillation for Binarization-aware Training

When the neural network is binarized, its representation capability is extremely limited, and the accuracy is significantly decreased compared with the full-precision counterparts. As shown in Figure 2, binarization makes the intermediate representation visually monotonous since it restricts the pixels to binary, while the original feature information is richer and contains much detailed information. However, since it is highly discrete in binarized representations, values on the edges falling to 1 or -1 outline the object, conveying more

(b) after applying wavelet transform

(Bℓai ⊗Bpℓt-is1

)

αℓai

(3)

(a) original

Figure 2: Compared with the original full-precision representation, the relative wavelet energy of the high-frequency component of the binarized representation is larger while that of the low-frequency component is much smaller.

important information than plain blocks. While it is wellknown that edges often present as local maxima of the gradient, which is hard to be improved directly by global optimization.

Fortunately, we find that the essence of the information inclination to edge is that the essential binarized representation tends to concentrate on high-frequency components. We use 2D Haar Wavelet Transform (WT) [Meyer, 1992], which is most frequently used as a separable transform that isolates horizontal and vertical edges, to decompose the representation into low and high-frequency components. The hidden state H inputted to a specific layer can be represented as a weighted sum of the wavelet function family:

where ϕ is the mother wavelet function with a specific time parameter, j = -1, · · · ,-N is the resolution level, and k determines the translation of the waveform.

To measure the information amount conveyed by the single component of representation, the relative wavelet energy is used to define the amount of information [Rosso et al., 2001]. The wavelet energy Ej at j-th level is first calculated as:

When we obtain low and high-frequency coefficients (Cj ∈ {CL,CH}, N = 2) by once decomposition, their relative wavelet energy pH and pL can be expressed as:

The larger relative wavelet energy demonstrates the information is more gathered in this component. As Figure 2 shows, compared with the full-precision representation, the relative wavelet energy of the high-frequency components of the binarized ones increases significantly, which implies that binarized representation inclines higher-frequency components.

Based on the above analysis, we propose a High-frequency Enhancement Distillation for binarization-aware training.

The scheme utilizes a pre-trained full-precision D-FSMN as the teacher, and enhances the high-frequency components of its hidden states during distillation. Specifically, we apply wavelet transform on the original representations, remove the low-frequency component, and then apply Inverse Wavelet Transform (IWT) function on the high-frequency components to retrieve selected representation. The process can be formulated as follow:

And then the emphasized high-frequency representations are added to the original ones:

where σ(·) is the standard deviation. Then, inspired by [Martinez et al., 2020], we minimize the attention distillation loss between ˆHT from the teacher and ˆHS directly from the hidden states of the student, which is expressed as:

where ℓ denotes the ℓ-th block and ∥ · ∥ is the L2-norm.

The HED scheme above makes it easier for the binarized student network to exploit the essential information from emphasized full-precision representations and improve accuracy.

3.3 Thinnable Binarization Architecture for Runtime Accuracy-efficiency Trade-off

As discussed earlier, binarization is an efficient compression approach with extremely lightweight 1-bit parameters and efficient bitwise operations, enabling fast inferences on resource-limited devices. As one of the advantages, binarization will not affect the model architecture, which is critical for well-designed structures. However, the energy budget varies between devices and even during wake-up and power-saving modes on a single device. Therefore, a lightweight yet adaptive binarized architecture, which can switch among different widths at runtime, permits instant and adaptive accuracyefficiency trade-offs for KWS.

Algorithm 1: The training process of our BiFSMN.

Input: Fixed pre-trained full-precision teacher MFP32 and thinnable binarized model M (BiFSMN) with N basic binarized blocks, training iterations T .

Output: Well-trained thinnable binarized model M

for all t = 1, 2, . . . , T do Forward propagate MFP(x) and obtain the information-enhanced intermediate features ˆHT = { ˆH1T , ˆH2T , . . . , ˆHNT }; for all δ = 1, 2, . . . , log2 N do Forward propagate M(x; δ) and obtain the intermediate features HδS = {HδS , H2δ S , . . . ,HNS }; Compute the distillation loss Lδdist by Eq. (14); Compute the cross-entropy loss LδCE; Descend Ltot as Eq. (15) and update M; Get the well-trained BiFSMNmodel M; Evaluate the BiFSMN on test dataset and get the accuracy.

We present a Thinnable Binarization Architecture (TBA) for KWS, which can select a thinner model with fewer layers at runtime, directly reducing the computational consumption. When we focus on the computational expensive backbone, the basic binarization architecture M′ containing N blocks (N ∈ {2n , n ∈ Z+}) can be expressed as:

where M′ and φℓ are the binarized network and ℓ-th binarized D-FSMN block, respectively, and x is the input of network. The TBA derived from M can be defined as:

where δ is the interval of selected layers, which is confined to be divisible into N. And each thinnable block Φℓ can be defined as

And the batch normalization in the function f(·) in Eq. (4) in ℓ-th binarized D-FSMN block φℓ are preassigned according to different variants in the thinnable network. The thinnable network architecture will skip intermediate blocks every δ layers by replacing them with identity functions. Figure 1 shows the formalization of our thinnable binarization architecture, and we also provide an instance for N = 8, δ = 1, 2, 4 in Figure 3, which is also the default setting in our experiments.

To optimize the binarization-aware training for the proposed TBA, we adopt the uniform layer mapping strategy to better align and learn representation in the HED:

The gradients from different switches are accumulated during backward propagation to update the weight jointly. According to the compression ratio in thinnable architecture, the weighted loss can be calculated as:

Ltot = ∑

δ

where LδCE denotes the cross-entropy loss of M(·; δ) and γ is a hyperparameter to control distillation impact, set to 0.01 as default. The detailed training procedures for the BiFSMN are listed in Algorithm 1.

3.4 Fast Bitwise Computation Kernel for Efficient Hardware Deployment

Benefiting from binarized weights and activations compressed to 1 32 of the original bit-width, a single binarized layer has an extreme-high 64× theoretical reduction of FLOPs [Liu et al., 2018]. However, when we implement and deploy the entire binary neural network on real-world hardware using existing binarization deployment frameworks, such as daBNN [Zhang et al., 2019] and Bolt [huawei noah, 2021], its overall inference efficiency is often significantly lower than the theoretical upper limit. One of the key bottlenecks of acceleration is the Binarized General Matrix Multiply (BGEMM) performed with the bitwise XNOR and Bitcount. Therefore, for efficient deployment on edge devices with limited computational resources, we further optimize the 1-bit computation with new instruction and register allocation strategy to accelerate the inference on ARMv8-A architecture widely used on edge devices. We dub it Fast Bitwise Computation Kernel (FBCK).

According to the number of registers on ARMv8 architecture, we first reallocate the registers in the kernel as five partitions in order to improve the register utilization and reduce memory footprint: partition A has four registers (except register v0) for one input (weight/activation), B has two for the other input, C has eight for intermediate results of EOR and CNT, D has eight for the output in one loop, and E has eight for the final results. Each input is packed as INT16. Each register in A stores one input while repeated 8 times, while each in B stores 8 different inputs. We first apply EOR and CNT for A with one register of B to get 32 INT8 results in intermediate partition C, and then perform ADD to accumulate the INT8 to D, and do the same for the other register of B. After sixteen times loop, we finally accumulate the INT8 data

stored in D to an INT16 register (in E) using long instruction

ADALP, which extends INT8 data to double width. FBCK makes full use of registers almost without idle bits during the computation. Refer to Figure 4 as illustration.

4 Experiments

In this section, we conduct experiments on the Google Speech Commands V1 and V2 datasets [Warden, 2018] to verify the effectiveness of BiFSMN and compare it with state-of-the-art (SOTA) binarization methods and various architectures.

4.1 Ablation Study

We perform ablation studies to investigate the effect of components of the proposed BiFSMN, including the Highfrequency Enhancement Distillation (HED) and Thinnable

(15)

Binarization Architecture (TBA), on the Speech Commands V1-12 and V2-12 KWS tasks.

As shown in Table 1, the vanilla binarization baseline suffers a significant performance drop in both datasets. The naive distillation scheme helps with the accuracy on the basic D-FSMN architecture, and the application of HED further improves the performance considerably based on distillation. It demonstrates that emphasizing high-frequency information makes it easier for the binarized network to exploit the most crucial representation. And the distillation strategy is also indispensable for better aligning and transferring information.

On the other hand, when solely utilizing TBA, we train a single model but optimize it in different parameter scales through weighted backward propagation. It shows a great deal of potential in not only the adaptive and lightweight computation at runtime but also the optimization of the binarized network during training. Moreover, jointly using HED and TBA further close the accuracy gap between the binarized model and the full-precision counterpart, which is less than 3% on both datasets.

4.2 Comparative Experiments

We first compare our BiFSMN with existing structureindependent binarization methods, including BNN [Courbariaux et al., 2016], DoReFa [Zhou et al., 2016], XNOR [Rastegari et al., 2016], Bi-Real [Liu et al., 2018], IR-Net [Qin et al., 2020], and RAD [Ding et al., 2019]. We evaluate these binarization methods on 8-block D-FSMN architecture with the same size as the largest variant of BiFSMN. The results in Table 2 show that our 1-bit BiFSMN completely outperforms other SOTA binarization methods by a wide margin. It is noteworthy that BiFSMN even enjoys competitive accuracy to full-precision counterparts within 4% average accuracy drop on both datasets.

Second, to validate the advantage of our TBA from the architecture perspective, we also compare it with various networks widely used in KWS, including FSMN [Zhang et al., 2015], VGG19bn [Simonyan and Zisserman, 2014], BCResNet [Kim et al., 2021], and Audiomer [Sahu et al., 2021]. We binarized these architectures with XNOR and IR-Net. In Table 3, our HED can generally be applied in FSMN-based architectures and make a difference for the binarized model performance. Moreover, equipped with TBA, BiFSMN can further strike a balance between accuracy and efficiency at runtime. We further prune the model width and provide an extremely tiny BiFSMNS (with 32 backbone memory size and 64 hidden size) with only 0.05M parameters and 9.16M FLOPs, demonstrating that our methods also works well on tiny networks.

4.3 Deployment Efficiency

To validate the practicability of BiFSMN, we test the actual speed of BiFSMN on Raspberry Pi 3B+ with 1.2GHz 64-bit ARMv8 CPU Cortex-A53.

According to Figure 5, due to the proposed optimized 1bit Fast Bitwise Computation Kernel, our BiFSMN delivers 10.9× acceleration compared to the full-precision counterparts. It is also much faster than the existing opensource high-performance binarization frameworks (daBNN and Bolt). Furthermore, benefiting from the thinnable architecture, BiFSMN can adaptively balance accuracy and efficiency at runtime according to the resources on device, and switches to BiFSMN0.5× or BiFSMN0.25× for further 15.5× and 22.3× speedups, respectively. It shows that our BiFSMN can satisfy different resource constraints.

IR-Net

XNOR

1/1

0.35

619.03 65.94

1/1

0.35

541.70 66.51

FSMN

D-FSMN

Full Prec.

HED (ours)

32/32

1/1

0.80 18577.49 98.95

0.80

1449.41 62.44

0.80

2567.19 62.44

0.45

1625.29 97.52

0.45

80.61 55.89

0.45

0.45

64.36 89.90

0.60

64.36 88.35

710.15 97.93

0.60

45.04 82.74

0.60

0.60

40.46 93.47

40.46 86.81

BiFSMN

[1,0.5,0.25]×

Full Prec.

32/32

0.60

710.15 97.93

IR-Net

1/1

0.61

HED (ours)

1/1

0.61

40.46 88.37

29.90 87.11

24.62 86.08

40.46 95.03

29.90 94.87

Speech Commands V1-12 task. #Param denotes million parameters.

5 Conclusion

We present BiFSMN, an accurate and extreme-efficient binary neural network for KWS. We first construct an HED scheme to emphasize high-frequency information to optimize the training of the binarized network. We also propose a TBA to achieve instant and adaptive accuracy-efficiency trade-offs at runtime. BiFSMN outperforms existing binarization methods by convincing margins and is even comparable to the full-precision counterpart. Moreover, our implementation for BiFSMN on ARMv8 real-world devices achieves an impressive 22.3× speedup and 15.5× storage-saving.

Acknowledgements

This work was supported in part by National Natural Science Foundation of China under Grant 62022009 and Grant 61872021, Beijing Nova Program of Science and Technology under Grant Z191100001119050.

606/44

dabnn

FP 377,

8L 81,8

2L 28,7

88517

11054

2,950,

Our fas

8L 34,6

4L 22,8

2L 16,9

BiFSMN

8L 81.9

4L 46,4

2L 28.8

bolt

8L(2,95

4L 26,2

2L(2,95

存储

2L 60K

8L 110

FP 1.7