Predicting Spectral Information for Self-Supervised Signal Classification

1 Introduction

Communication signal classification techniques have emerged as a powerful tool in a variety of important applications, such as the detection of Internet of Things attacks [Huang et al., 2021], 5G and advanced non-cooperative communication [Hermawan et al., 2020], as well as spectrum sensing and electronic warfare [Yakkati et al., 2021]. With the continuous advancement of Deep Learning (DL), DLbased communication signal classification algorithms have significantly outperformed traditional algorithms in recent years [O'Shea et al., 2016]. However, in some real-world situations, especially for time series data, obtaining labeled data is not only a time-consuming and expensive process but also requires substantial expert knowledge. Therefore, exploring how to extract features from unlabeled data is a direction worth investigating.

∗Corresponding author

Self-Supervised Learning (SSL) is a form of unsupervised learning that leverages automatically generated pretext tasks to extract valuable supervisory signals from unlabeled data. These pretext tasks are challenges designed by the model itself, and the model learns valuable representations for downstream tasks by solving these challenges. Unlike supervised learning, which requires a large amount of manually annotated data, SSL can learn valuable representations from unlabeled data by exploiting the inherent characteristics of the data, thereby avoiding the tedious and expensive process of manual data annotation. This approach fully exploits the potential information in the data, providing good initial feature representations for downstream tasks, which can improve the performance and generalization ability of the model. SSL has recently achieved great success in the fields of Computer Vision (CV) [Jing and Tian, 2020] and Time Series (TS) [Zhang et al., 2024]. Consequently, applying SSL to communication signal field is a promising research direction that is worth further exploration.

However, in current research on SSL for communication signal classification, several core challenges remain to be addressed. One major challenge is that, compared to general image or time series data, communication signal data is more heavily reliant on spectral analysis [Zeng et al., 2019]. The complex time-frequency characteristics and relationships of these signals require models to efficiently capture both timedomain and frequency-domain information simultaneously. However, existing SSL methods often borrow pretext tasks from CV or TS domains, neglecting the unique modulation semantics and spectral features of communication signals. This oversight makes it difficult for the models to accurately understand the inherent structure of the signals. Another significant challenge is that commonly used data augmentation techniques in other domains do not translate well to communication signals. For instance, transformations like rotation and cropping in CV overlook the semantic information of time series data and may disrupt the temporal dependencies in communication signals, such as disturbing the peaks and valleys of the waveform. In the TS domain, data augmentation techniques are typically designed based on the subsequence consistency assumption [Franceschi et al., 2019]. However, since communication signal data is fundamentally built on baseband modulation code sequences to carry information, the sub-sequences in such data often contain incomplete or biased semantic information between them, which makes these methods less effective for communication signals. These challenges have driven us to seek a novel selfsupervised training framework that does not rely on traditional data augmentation, aiming to more effectively model and utilize the time-frequency characteristics of communication signals.

Communication modulation signals contain a wealth of exploitable spectral information with modulation semantics. The extraction of frequency domain features has long been a focus in the field of signal processing [Katsaggelos et al., 1993]. The different modulation schemes exhibit distinct spectral feature distributions in the frequency domain. Figure 1 shows an example of this phenomenon. The spectral energy of the 8PSK modulation scheme is primarily concentrated in the high-frequency region, while the spectral energy of CPFSK is mainly distributed in the mid to lowfrequency range. Inspired by this, we designed a novel selfsupervised training strategy SGSSC that enables the model to learn cross-domain data transformation from time domain to frequency domain. This training strategy allows the model to develop the ability to extract distribution of spectral information with modulation semantics from time-domain signals. To our knowledge, this is a previously unexplored pretext task that is more suitable for communication signal data formats rich in spectral information, compared to generative [He et al., 2022] or contrastive [He et al., 2020] SSL methods. Our approach innovatively uses spectral information to guide the model in learning implicit frequency domain knowledge structures from time-domain data, thereby building a cross-domain knowledge bridge. Additionally, to enhance the model's temporal modeling capability, we employ a masking strategy on the communication signal data within the pretext task. Concurrently, we also recognize that the spectral information inherently carries certain modulation semantics that are closely related to the downstream classification task. Applying a traditional fine-tuning approach on the downstream task may potentially lead to the loss of valuable pre-training knowledge. To address this challenge, we have designed a novel fine-tuning strategy based on attention mechanisms, which enables our model to adaptively integrate different levels of pre-trained features. Our contributions can be summarized as follows:

• We propose a self-supervised training method SGSSC specifically designed for communication signal data, which leverages the spectral information containing modulation semantics in the signals. This approach enables the model to extract deeper semantic features from time-domain data for downstream tasks.
• We introduce a novel fine-tuning strategy based on attention mechanisms for downstream tasks, enabling the adaptive integration of pre-trained features from various levels, thereby preserving valuable knowledge learned during the self-supervised training phase.
• We validated the effectiveness of this self-supervised training framework in downstream communication signal classification tasks. When the proportion of labeled samples is 0.5%, our method achieves an average improvement of 2.3% compared to the best-performing SSL method.

2 RelatedWork

2.1 Signal Classification

Signal classification has long been a pioneering problem in the field of communication. Traditional modulation signal recognition methods rely on manual feature extraction, including spectral features, instantaneous parameter statistical features, and higher-order cumulants [Zhang et al., 2001], as well as likelihood ratio methods for classification tasks [Xu et al., 2011]. However, these methods heavily depend on expert design and signal conditions. With the remarkable achievements of deep learning, a wave of DL-based models has been applied to signal classification. Intuitively, the adopted strategy is to transform signals into a visual format to leverage CV-based models for classification, such as CNN [HuynhThe et al., 2020], RNN [Hong et al., 2017], and Transformer [Hamidi-Rad and Jain, 2021] architectures. The effectiveness of DL-based communication signal classification methods hinges on the availability of a large amount of labeled data. However, in the communication signal field, obtaining labeled data is more challenging compared to the CV field, especially in non-cooperative communication scenarios that require strong expert knowledge. Therefore, correctly handling unlabeled modulation signals is key to successfully completing classification tasks.

2.2 Self-Supervised Learning

SSL primarily leverages pretext tasks to extract supervisory signals from large-scale unlabeled data. In the field of CV, it can be categorized into two types: generativebased methods and contrastive-based methods. The main idea behind generative-based methods is to learn visual features through image generation tasks, such as grayscale image colorization [Zhang et al., 2016], image inpainting [Pathak et al., 2016], and playing image puzzles [Noroozi and Favaro, 2016]. Contrastive-based methods, on the other hand, focus on modeling the relationships between different instances using a simple discriminative structure, examples include SimSiam [Chen and He, 2021], SimCLR [Chen et al., 2020a], and SWAV [Caron et al., 2020a]. Due to the success of contrastive learning methods in the field of CV, some researchers have also introduced them into the field of communication signals [Kong et al., 2023]. However, these methods primarily apply the original algorithmic processes from the CV field, merely altering the model structures or data augmentation methods. They lacked algorithmic innovations tailored to the unique data formats of communication signals or the incorporation of expert knowledge specific to this field.

3 Method

In this section, we will provide an overview of the selfsupervised training framework we have proposed. First, we will introduce the pretext task we have designed specifically for communication signal data. Then, we will elaborate on the masking strategy implemented to further enhance the model's temporal modeling capability for communication signals. Finally, we will present the attention-based finetuning method developed to integrate pre-trained features for downstream tasks. Figure 2 illustrates the complete pipeline of the SGSSC method.

3.1 Pretext Task

The Discrete Fourier Transform (DFT) can be used to analyze the frequency characteristics of signals [Fonseca Guerra et al., 1998], such as identifying the main frequency components in a signal. This is widely applied in fields like audio signal processing and communication system analysis. We leverage the frequency components obtained through the DFT of the time-domain signal x to model Xgt = [X [0] ,X [1] , . . . ,X [L-1]]. Specifically, the p-th component X [p] of the DFT of x is given by the following equation:

where x [k] is the k-th signal value of time-domain signal x. Due to the properties of the DFT, the resulting frequencydomain representation Xgt is a complex vector with the same dimensions as the input time-domain signal x.

Different modulation schemes exhibit distinctly different characteristics in the frequency domain, while signals employing the same modulation schemes tend to have remarkably similar spectral information. We believe that the correlation between modulation schemes and spectral information serves as a powerful prior knowledge, enabling models to better comprehend the inherent modulation semantics of communication signals. Therefore, we designed a simple yet ingenious pretext task. We use the time-domain signal as the input to the model, while the spectrum obtained through the DFT serves as the ground truth to guide the model's training:

where ˜X is the output spectrum predicted by the model. We use Mean Squared Error loss to optimize the model for iterative updates.

By designing a pretext task that encourages the model to learn cross-domain transformations, we enable the model to uncover the intrinsic relationships between the temporal structure of the input signal and the spectral information imbued with modulation semantics. This allows the model to associate its feature representations with the underlying modulation characteristics of the signal. Consequently, in downstream tasks, this holistic understanding empowers the model to extract more fundamental and higher-level features from the time-domain signals, thereby reducing the model's reliance on large amounts of labeled data.

3.2 Masking

In the field of signal classification, enabling models to understand temporal relationships has long been a focus of research. Some scholars have approached this by modifying model architectures, incorporating LSTM [Emam et al., 2020] or Transformer [Cai et al., 2022] modules into models. While this approach equips models with the ability to model the temporal properties of signals, it often lacks a robust loss function to constrain the modeling process. In the fields of CV/NLP, masked autoencoders have proven to be an effective SSL method [Doersch et al., 2015]. This strategy of predicting masked portions allows models to better understand the contextual relationships within data. Inspired by this, we have also introduced the strategy of masking. Specifically, we randomly set certain signal values in the time domain to zero at a certain ratio, and then input these masked signals into the model for pretext task training. Through this approach, the model is forced to leverage the surrounding signal values to infer and reconstruct the masked information. This process encourages the model to develop a more profound understanding of the underlying temporal structure and dependencies within the communication signals.

3.3 Fine-Tuning of AttentionMechanisms

After completing self-supervised training, it is generally believed that the shallow layers of the model capture lowlevel features, while the deeper layers learn features that are strongly related to the pretext tasks [Zhou et al., 2018]. In the field of CV, the pretext tasks often have significant differences from the downstream tasks. Therefore, during finetuning for downstream tasks, it is common practice to use the earlier layers of the complete model as feature extractors [Jing and Tian, 2020], discarding the latter layers of the model structure. However, the SGSSC method differs in this regard. The spectrum of a signal itself possesses inherent modulation semantics, which can be leveraged as features in downstream classification tasks. Consequently, the features at the backend of the model obtained through self-supervised training are closely related to the downstream tasks. Discarding these features would hinder the effective utilization of the knowledge acquired during self-supervised training when fine-tuning for downstream tasks. To address this issue, we design an attention-based fine-tuning approach to integrate features from different layers of the model.

The Figure 2c illustrates the detailed structure of the attention module. In principle, the concept of this module can be embedded in any model architecture. In our case, we primarily integrate the module into the transformer architecture. The model structure we use is similar to the Vision Transformer (ViT) [Dosovitskiy et al., 2020], with simple modifications made to accommodate the data format of signals.

The output of the n-th transformer block in the model can be defined as Fn ∈ R(M×L) , where M is the number of tokens and L is the feature length of each token. The model consists of a total of N transformer blocks. During finetuning, we first concatenate the output features from each layer along the token dimension, resulting in a richer feature representation F ∈ R((N∗M)×L) . This feature incorporates information from different layers, encompassing both low-level features of the signals and high-level features related to the pretext tasks. Next, we employ a token-level attention mechanism to process this feature. There are two main reasons for this approach: firstly, the dimensionality of the feature is quite large, and using it directly for fine-tuning on downstream tasks may easily lead to overfitting, necessitating dimensionality reduction; secondly, the use of tokenlevel attention allows us to selectively focus on and integrate features across different layers and tokens, better leveraging the time-frequency understanding developed during the selfsupervised training phase. This enables the model to seamlessly transfer the learned knowledge to the downstream signal classification task, thereby improving performance. Our specific approach is outlined as follows:

where AvgPool is the average pooling layer. MLP is the fully connected layer, and σ is the sigmoid activation function, which maps the features to values between 0 and 1. The variable A is the attention scores we aim to obtain. Finally, as shown in Equation (4), we multiply each token by its corresponding attention score and then aggregate the results, which serves as the feature representation for the downstream task:

where ˜ F is the input feature for the downstream task.

4 Experiment

4.1 Dataset

We adopted the data generation methodology used in the publicly available RML2016.10a dataset [O'shea and West, 2016] to create datasets under three distinct channel conditions: Additive White Gaussian Noise (AWGN) channel, Rayleigh (Ray) channel, and Rician (Ri) channel. The AWGN channel has additive white Gaussian noise, with the amplitude following a Gaussian distribution and a constant power spectral density function. The Rayleigh channel is used to model scenarios where there is no direct link between the transmitter and receiver, and the signal reaches the receiver through reflection or diffraction. The Rician channel is used to model channel fading situations where there is a direct path and multiple reflected paths simultaneously. Each dataset contains 220,000 modulated signals, comprising

11 modulation schemes, with 20,000 signals per modulation class. The signals are generated at 20 different signal-to-noise ratios (SNR), ranging from -20 dB to 18 dB with a step size of 2 dB. Each class and SNR combination has 1,000 signals.

We further evaluated the effectiveness of our method using six time series datasets: Human Activity Recognition (HAR) [Anguita et al., 2013], Epilepsy Seizure Prediction (ESP) [Andrzejak et al., 2001], and several datasets from the UCR Repository [Dau et al., 2019], including Wafer, PhalangesOutlinesCorrect (POC), ProximalPhalanxOutlineCorrect (PPOC), and StarLightCurves (SLC).

For all datasets, we performed a split, with 60% used for self-supervised training, 20% for validation, and 20% for testing. During fine-tuning on downstream tasks, we randomly sampled the corresponding proportion of data from the selfsupervised training set and utilized their labels.

4.2 Implementation Details

To validate the effectiveness of our method, we followed a standard linear evaluation scheme [Chen et al., 2020b]. In this scheme, a linear classifier (a single fully connected layer) is trained on a frozen self-supervised pre-trained model. In the self-supervised training experiment, we used a mask ratio of 0.7, the Adam optimizer with a learning rate of 0.0001, a batch size of 64, and trained for a total of 500 epochs. For the fine-tuning experiments on downstream tasks, we employed the Adam optimizer, setting the learning rate for the classifier to 0.06 and a batch size of 64. Our model architecture is similar to ViT, with a patch size of 1×16, 8 layers, a hidden size of 128, an MLP size of 1024, and 8 attention heads. All experiments were conducted on a GeForce RTX 3090, and the reported results represent the average performance over five independent runs.

4.3 Overall Performance

To validate the effectiveness of the proposed self-supervised pre-training method SGSSC, we fine-tuned the model on downstream classification tasks using 0.1% and 0.5% of the labeled data, with accuracy as the evaluation metric. The results are shown in Table 1. The experimental datasets encompass various scenarios, including single-channel and cross-channel settings. For instance, "AWGN" indicates both self-supervised training and downstream task fine-tuning are performed on the AWGN dataset, whereas "A → Ray" denotes that self-supervised training is conducted on the AWGN dataset and the downstream classification task is fine-tuned on the Rayleigh dataset. This setup allows us to assess the method's performance on in-distribution tasks as well as its generalization capability in cross-distribution transfer tasks.

To compare performance, we selected 13 comparative methods, categorized into three groups: the baselines include Random Init and Supervised, where Random Init represents training a linear classifier on a frozen and randomly initialized encoder, and Supervised refers to fine-tuning the entire model. DCL [Chuang et al., 2020], NNCLR [Dwibedi et al., 2021], SimCLR [Chen et al., 2020a], SimSiam [Chen and He, 2021], SWAV [Caron et al., 2020b], and MAE [He et al., 2022] are current mainstream self-supervised learning methods in the computer vision domain. SSL-ECG [Sarkar and Etemad, 2020], CPC [Oord et al., 2018], TS-TCC [Eldele et al., 2023], InfoTS [Luo et al., 2023], and TimesURL [Liu and Chen, 2024] are self-supervised learning methods specifically designed for time series data. Additionally, to ensure fairness, we designed a control version SGSSC* that does not employ our proposed attention-based fine-tuning strategy, to validate the pure effectiveness of the feature representations. As shown in Table 1, the proposed SGSSC and SGSSC* methods

(a) 0.1% of labeled data

(b) 0.5% of labeled data

achieve superior performance across all experimental scenarios.

Furthermore, to demonstrate the potential of our method on time series data, we conducted experiments on six time series datasets. For these datasets, we fine-tuned the model using 1% and 5% of the labeled data. The experimental results, presented in Table 2, indicate that our method outperforms others on five of these datasets as well as in terms of the average accuracy across all datasets.

4.4 Mask Ratio

To investigate the impact of different mask ratios in selfsupervised training on downstream task performance, we set mask ratio to 0.1, 0.3, 0.5, and 0.7, and conducted selfsupervised training on the AWGN dataset. Figure 3 shows the frequency-domain reconstruction quality of a BPSK modulation scheme with a SNR of 18 dB under different mask ratios. As observed, increasing the mask ratio leads to degradation in reconstruction details. Nevertheless, the overall frequency distribution characteristics are still well preserved.

Figure 4 presents the experimental results of models trained with different mask ratio on downstream tasks. It is evident that when the mask ratio is 0.7, the model demonstrates excellent feature extraction capabilities. This is because the signal data exhibits strong temporal continuity with minimal fluctuations. At lower mask ratios, the model tends to learn shortcuts in temporal modeling, while only at higher mask ratios is the model compelled to discover more robust features within the signal, thus facilitating the effective transfer of knowledge to the downstream tasks.

4.5 Visualization and Computational Complexity

To more intuitively demonstrate the allocation of feature weights by our proposed attention-based fine-tuning strategy, we visualized the attention weights across different layers of the model, with the results presented in Figure 5. In the visualization, the horizontal axis represents the attention weights of different tokens in the model arranged in the order of forward propagation layers, while the vertical axis corresponds to datasets from three channel environments. From the figure, it can be observed that tokens from deeper layers exhibit significantly higher attention values, whereas those from shallower layers have relatively lower attention values. This aligns with the objective of our proposed fine-tuning strategy, indicating that the token-level attention mechanism can selectively focus on high-level features that are closely related between the pretext task and downstream task, while retaining low-level features from shallow layers to provide foundational information.

Meanwhile, we also calculated the computational cost in FLOPs and the number of parameters for our method and the comparative methods discussed in this paper, with the results shown in Table 3. As seen in the table, our method has computational and parameter counts similar to some of the current mainstream approaches. This is because the pretext task we designed are very simple, and the attention-based fine-tuning strategy only adds a few small linear layers, resulting in minimal overhead.

4.6 Ablation Experiment

We conducted an ablation study on the attention mechanism and masking strategy using three datasets, with an experimental setting of 0.1% labeled data for downstream tasks. Table

4 presents the ablation results. The table shows that incorporating all components improves performance compared to the baseline. On the AWGN dataset, adding only the attention mechanism without masking the time-domain data leads to a 7.3% performance increase. Using a masking ratio of 0.7 on the time-domain data without the attention mechanism results in a 11.2% performance boost. When both modules are employed simultaneously, the model's performance increases by 16.3%. These findings indicate that both modules contribute significantly to the model's performance, facilitating efficient knowledge transfer to downstream tasks. The attention mechanism helps the model focus on features highly relevant to the downstream task, while the masking strategy encourages the model to learn more robust temporal modeling from partially masked time-domain signals.

5 Conclusion

Acquiring labels for communication signal data is often a challenging task. To address this issue, this paper explores a self-supervised learning method called Spectral-Guided SelfSupervised Signal Classification, which is particularly suited for time series data with rich spectral information. By leveraging spectral information, this method facilitates the model's learning of more prior knowledge with modulation semantics. To validate its effectiveness, we conducted experiments using both signal and time series datasets, demonstrating that this training approach aids the model in acquiring richer feature representations and achieving superior performance in downstream tasks. We hope that this self-supervised learning method will advance research across various time series domains.