PhysioGMC: Generalizable Multi-modal Coordination for Physiological Signals

1 Introduction 1

Continuous monitoring of physiological signals plays an essential role in assessing human health in both clinical and daily-life settings. In clinical practice, signals such as electrocardiography (ECG), arterial blood pressure (ABP), and polysomnography (PSG) are routinely collected for tasks including arrhythmia detection, hypertension management, and

1 Code and appendix: https://github.com/nott1999/PhysioGMC

(a) Clinical Settings

Figure 1: Illustration of physiological signal acquisition in (a) clinical settings using ECG and blood pressure measurements, and (b) daily-life settings using continuous PPG monitoring with wearables.

sleep staging. With the widespread adoption of wearable devices such as smartwatches, wearable signals, such as PPG [Hwang et al., 2021], have emerged as scalable, non-invasive alternatives for continuous health monitoring in everyday life. Together, these developments have led to the rapid growth in the amount of physiological data collected across diverse environments.

The growing volume of physiological data collected both within and beyond clinical settings has made manual expert review increasingly impractical, motivating deep-learningbased automated analysis systems. Recent advances have achieved strong performance in health monitoring tasks, including atrial fibrillation (AF) detection [Torres-Soto and Ashley, 2020; Das et al., 2022], hypertension assessment [Leitner et al., 2022; BN et al., 2024], sleep apnea detection [Olsen et al., 2024; Sharan et al., 2020], and sleep staging [Kotzen et al., 2023]. Despite these successes, most existing methods rely on a single physiological modality, thereby overlooking the complementary information available across multiple physiological signals [Gil et al., 2010; Mukkamala and Hahn, 2018]. This limitation is particularly important because physiological modalities differ substantially in their reliability, accessibility, and suitability for long-term deployment.

Physiological signals collected in clinical settings, such as ECG, ABP, and PSG, provide more reliable, accurate, and clinically interpretable measurements of human health compared to signals measured by wearable devices. However, measuring these signals requires specialized and cumbersome equipment, making long-term continuous monitoring in nonclinical environments impractical, uncomfortable, and costly, as illustrated in Fig. 1(a). In contrast, wearable devices enable more feasible and user-friendly long-term health monitoring through non-invasive measurements, as shown in Fig. 1(b). This shows a trade-off between the fidelity of clinical-grade signals and the practicality of continuous real-world monitoring. To balance this trade-off, it is desirable to complement the strengths of multimodal physiological modalities by leveraging their inherent relationships to enhance the utility of wearable signals in real-world application scenarios.

Although training with multimodal data has been shown to improve the performance of unimodal models [Zadeh et al., 2020; Ma et al., 2021; Liu et al., 2023; Wu et al., 2024], this paradigm is understudied in the context of physiological sensing. Existing approaches in this domain generally fall into two categories: 1) inferring the absent modalities during testing from the available ones; 2) aligning features across modalities to achieve multimodal coordination. Inference-based methods [Tian et al., 2023; Vo et al., 2024; Ji and Zhou, 2024] propose reconstructing absent physiological modalities from available modalities using generative models and subsequently applying the inferred signals to downstream tasks. However, such approaches are sensitive to reconstruction errors and may introduce additional noise that degrades performance. Recently, some work has focused on achieving multimodal coordination for physiological signals. SiamAF [Guo et al., 2023] employs a Siamese network to learn shared features between ECG and PPG signals, thus improving AF detection accuracy using only PPG at inference. But these existing methods overlook the substantial crosssubject differences inherent in physiological signals, hindering their generalization to unseen subjects.

In this work, we propose PhysioGMC, a multimodal coordination framework designed for physiological signals, which coordinates wearable modalities with clinical-grade modalities during training to improve cross-subject performance of a single wearable modality at test time. During training, PhysioGMC employs a cross-modal contrastive learning module composed of two contrastive losses to align subject-agnostic, label-decisive features across clinical-grade and wearablegrade physiological modalities. The self-supervised crossmodal contrastive loss encourages consistency across features learned from different modalities of the same input data, while the supervised cross-modal contrastive loss learns label-decisive features agnostic to subjects. At test time, PhysioGMC achieves superior performance using only the wearable PPG signal, thereby bridging the gap between the benefits of multimodal learning and the practical constraints of unimodal wearable deployment.

To the best of our knowledge, we introduce the first generalizable multimodal coordination framework for physiological signals. Extensive experiments on multiple public datasets demonstrate that our method consistently outperforms existing approaches across several health monitoring tasks, achieving superior cross-subject performance using only a wearable physiological signal, PPG, during inference.

2 RelatedWork

2.1 Inter-subject Variability in Physio. Signals

Physiological signals exhibit substantial inter-subject variability, which limits the generalization of models trained on a fixed set of subjects to unseen individuals. Accordingly, recent studies have explored subject-invariant representation learning for physiological signals. For example, some works [Hwang et al., 2021; Han et al., 2024] adopt adversarial training to disentangle task-related, subject-invariant features from subject-specific features and noises, thereby mitigating inter-subject variability. Another work [Kim et al., 2024] leverages both deep metric learning and contrastive learning strategies to learn subject-invariant features. While these approaches improve inter-subject generalization, they typically consider each physiological signal in isolation and therefore do not exploit complementary information across modalities.

2.2 Multimodal Coordination for Physio. Signals

Previous studies [Zadeh et al., 2020; Rahate et al., 2022] have shown that models trained collaboratively using multimodal data, which is referred to as multimodal co-learning, can outperform those trained solely on unimodal data in downstream unimodal tasks. Motivated by these works, researchers explore multimodal co-learning for physiological signals. MERL [Liu et al., 2024] proposes a cross-modal contrastive learning strategy to align ECG features with text features during the pretraining of an ECG-based encoder. SleepSMC [Ma et al., 2025] improves sleep staging performance when only one modality is present at the testing stage by introducing modality-level contrastive coordination mechanisms. However, existing multimodal co-learning approaches largely overlook the inter-subject variability intrinsic to physiological signals, which limits their effectiveness and motivates our generalizable multimodal coordination framework, PhysioGMC.

2.3 Multimodal Domain Generalization (DG)

Most existing research for DG focuses on learning domaininvariant features for unimodal data. RNA-Net [Planamente et al., 2022] is among the first methods to learn generalized multimodal features across domains. SimMMDG [Dong et al., 2023] aims to achieve domain generalization in multimodal settings by splitting the representations into modalityspecific and modality-shared components. CMRF [Fan et al., 2024] proposes learning representations generalizable to unseen domains by constructing modality-consistent flat loss regions via cross-modal knowledge transfer. However, these works primarily focus on other modalities instead of physiological signals. However, these methods are primarily developed for general multimodal data and have not been investigated for physiological signals.

3 Feasibility of Multimodal Coordination for Physiological Signals

The practice of multimodal learning paradigms has demonstrated that models trained with multiple modalities can outperform those trained on a single modality, even if only one modality is available at deployment. Previous studies [Zadeh et al., 2020; Lu, 2023] attribute this advantage to the interrelated and complementary properties of the modalities used during training. Building on this insight, we refine the conditions introduced in [Zadeh et al., 2020] under which multimodal coordination improves upon unimodal learning.

Let FΘ denote a model parameterized by learnable parameters Θ. During training, the model has access to multimodal data {xi | i ∈ M} and the corresponding downstream-task label y, where M denotes the full set of modalities. At test time, only a subset of modalities, {xm | m ∈ S}, is available, where S ⊂ M. The set of modalities unavailable at test time is given by the complement Sc = M \ S.

Theorem 1. Assume that the modalities in M are mutually informative, i.e., (I ({xm | m ∈ S} ; {xn | n ∈ Sc }) > 0), and that incorporating additional modalities improves downstream task performance: (H (y | {xm | m ∈M}) < H (y | {xm | m ∈ S}) < H(y)), where I(·; ·) denotes mutual information and H(·) denotes entropy. Under these assumptions, there exist model parameters θ ∈ Θ such that,

The right-hand inequality in Eq. (1) holds if the following holds:

Although Zadeh et al. [Zadeh et al., 2020] identify Eq. (2) as a necessary condition for multimodal co-learning to improve unimodal performance, this condition alone is insufficient. Our extension, summarized in Eq. (3) in Theorem 1, demonstrates that, by capturing connections between modality sets {xm | m ∈ S} and {xn | n ∈ Sc } that preserve labelrelevant information, multimodal models can outperform unimodal counterparts, even when only a subset of modalities is available at inference, as stated in Eq. (1).

To empirically validate this principle in the context of physiological signals, we adopt the multimodal SimCLR [Chen et al., 2020] framework as a representative multimodal co-learning approach and apply it to the AF detection task. It enforces cross-modal alignment by treating temporally synchronized signal samples from different modalities as positive pairs, encouraging the learned label-decisive representations to encode shared information across modalities. Therefore, SimCLR instantiates the conditions for effective multimodal co-learning formalized in Theorem 1, making it an appropriate choice for empirically validating the benefits of multimodal co-learning over unimodal learning.

Despite improved performance over unimodal baselines, the results in Table 1 show that conventional multimodal coordination methods suffer from notable inter-subject variability and limited generalization to unseen subjects. This motivates our proposed framework, PhysioGMC, which collaboratively learns generalizable representations frommultimodal physiological signals.

4 ProposedMethod

We now provide an overview of the proposed PhysioGMC framework, as illustrated in Fig. 2. PhysioGMC consists of two primary components: 1) a unimodal feature extraction and classification backbone that learns representations from each physiological modality; and 2) a cross-modal contrastive learning module that combines a self-supervised contrastive loss for instance-level alignment across modalities with a supervised contrastive loss for aligning label-relevant features from the same class across modalities while separating them from features of other classes and subject-specific features.

4.1 Unimodal Feature Extraction and Classification

In the proposed framework, we adopt EfficientNet [Tan and Le, 2019] as the backbone encoder for extracting unimodal features. For a given input xmi of the modality m, the corresponding unimodal feature zm i is obtained as zm i = gm(xmi ), where gm(·) is the modality-specific feature encoder and i ∈ {1, · · · ,N} is the input index. The learned feature zm i is further decomposed into the label-relevant, subject-agnostic feature zm i,a and the subject-specific feature zm i,s using a feature decomposer d(·) These decomposed features are subsequently used for downstream classification.

The modality-specific classification loss is defined as:

where LCE denotes the cross-entropy (CE) loss, fm and f subj m are the modality-specific classifiers, yi is the label of downstream tasks, and ysubj i is the subject ID label. Minimizing the above classification loss encourages the model to learn the label-related features and the identity-related features for each modality separately.

4.2 Cross-modal Contrastive Learning

As discussed in Section 3, multimodal learning paradigms can outperform unimodal counterparts when they capture label-decisive mappings across different modalities. Motivated by this insight, we propose a cross-modal contrastive learning module comprising two objectives: a self-supervised contrastive loss and a supervised contrastive loss. This module learns subject-agnostic, label-relevant representations that are consistently aligned across modalities, thereby implicitly modeling label-discriminative relationships among different physiological signals.

For the contrastive learning, the extracted unimodal features of the modality m are normalized using L2 normalization as ˜ zm i,a = zm i,a ∥zm i,a ∥ , ˜ zm i,s = zm i,s ∥zm i,s ∥ . We assume that each batch of the data used for training comprises N quadruplets, {xi, yi, ysubj i }i=1,··· ,N, where xi = [xmi ; xki ] denotes temporally synchronized data instances from two physiological modalities m and k, and i is the input index.

Self-supervised Contrastive Loss

As the first component of the proposed module, we design a self-supervised cross-modal contrastive loss, defined in Eq. (5), that enforces alignment between representations of the same input data across modalities.

where τ is the temperature scaling parameter, controlling the sharpness of the similarity scores, and · is the inner product of two normalized features to measure their similarity.

The self-supervised contrastive loss encourages representations extracted from temporally paired modalities to be aligned in the embedding space, without relying on label supervision. Consequently, it enables the effective incorporation of large-scale unlabeled multimodal physiological data during training. An illustration of the self-supervised crossmodal contrastive loss is provided in Fig. 3(a).

Supervised Contrastive Loss

Although the self-supervised contrastive loss aligns features across modalities, the learned features may still contain labelirrelevant or subject-specific information. The entanglement of these two kinds of information can hinder discrimination and generalization of learned representations in downstream tasks, leading to degraded cross-subject performance. Thus, we propose a modified supervised contrastive loss to learn multimodal features that only contain subject-agnostic, labeldecisive information. This contrastive loss utilizes both task and subject label information as supervision, drawing features from the same class closer in the embedding space while pushing apart features from different classes, as illustrated in Fig. 3(b). This loss consists of two loss terms: the unimodal contrastive loss term and the cross-modal contrastive loss term.

To construct the supervised contrastive loss, we concatenate the normalized label-relevant subject-agnostic feature {˜ zm i,a }i=1,··· ,N and the normalized subject-specific feature {˜ zm i,s }i=1,··· ,N into a new feature vector z′m = [{˜ zm i,a }i=1,··· ,N ; { ˜ zm i,s }i=1,··· ,N]. The task label yi is assigned to˜ zm i,a while the subject ID label ysubj i is assigned to ˜ zm i,s . Consequently, we define the combined label vector as y′ = [{yi}i=1,··· ,N; {ysubj i }i=1,··· ,N]. The modified supervised contrastive loss is denoted as follows:

where i ∈ I ≡ {1, · · · , 2N} is the input index. We define P(i) ≡ {p : y′p = y′i } and A(i) ≡ I\{i}. N ′ is the cardinality of P(i). M is the set of available modalities during the training process. β1 is the hyperparameter that controls the relative importance of unimodal and cross-modal contrastive learning terms. τ and the similarity operator · have the same definitions as in the self-supervised contrastive loss.

In this supervised contrastive loss, the unimodal contrastive loss term encourages within-class consistency by attracting the same-class features within each modality. Complementarily, the cross-modal contrastive loss term encourages inter-modal consistency by aligning the features of the same class across modalities. Together, these terms guide the model to extract label-relevant information from multimodal data.

A supervised contrastive loss based solely on task labels may still preserve subject-specific information in the learned representations, thereby limiting generalization to unseen subjects. To mitigate this concern, we incorporate subjectspecific features ˜ zm i,s labeled with subject identities into the modified supervised contrastive loss. By treating the labelrelevant feature ˜ zm i,a and the subject-specific feature ˜ zm i,s extracted from the same input data as belonging to different classes, the proposed supervised contrastive loss effectively disentangles label-relevant information and subject-specific information, yielding label-relevant representations that are more generalizable across subjects.

4.3 Final Loss

The final loss function is defined as the weighted sum of the previously defined losses:

where λcls, λself , and λsup are the hyperparameters that control the relative importance of different loss terms.

5 Experiments and Results

5.1 Experimental Settings

Datasets. We conduct experiments on two subsets derived from the real-world ICU dataset MIMIC-III [Johnson et al., 2016], referred to as MIMIC-AF and MIMIC-Hyper, as well as on a subset of the MESA dataset [Chen et al., 2015], which includes 2237 participants enrolled in a sleep exam. Both MIMIC-AF and MIMIC-Hyper comprise physiological recordings from 300 critically ill subjects. The former subset contains ECG and PPG signals, while the latter contains ABP and PPG signals. All signals are segmented into 10-second windows for both ICU subsets. The MESA subset comprises overnight PSG recordings from 100 subjects, each containing synchronized 30-second ECG and PPG segments.

Downstream Tasks. We use two cardiovascular health monitoring tasks and one sleep monitoring task for evaluation over three datasets: AF detection on the MIMIC-AF, blood pressure classification on the MIMIC-Hyper, and sleep stage classification on the MESA subset. For the MIMICAF, each segment is labeled according to the corresponding ICD-9 diagnostic codes. For the MIMIC-Hyper, segments are categorized into four classes: normal, high-normal, Grade 1 hypertension, and Grades 2-3 hypertension. The categorization is based on their systolic blood pressure (SBP) and diastolic blood pressure (DBP) values, following the ESC/ESH guidelines [Mancia et al., 2014]. For the MESA subset, we consider a four-class sleep-staging task comprising wake, light sleep (N1/N2), deep sleep (N3), and Rapid Eye Movement(REM).

Evaluation Settings. To evaluate the generalization performance of PhysioGMC, we perform a five-fold cross-subject validation with no overlap between training and testing subjects. The final results are averaged across all five folds for an overall evaluation. We run each experiment with five random seeds to account for performance variability and report the mean and standard deviation of the five runs' results. Additionally, the Wilcoxon signed-rank test is used to assess the statistical significance of performance improvements achieved by the proposed PhysioGMC over baseline methods.

5.2 Experiment Results

To evaluate the proposed PhysioGMC, we compare it with eight baseline methods. All methods are evaluated under the setting where clinical ECG or ABP signals are available as auxiliary modalities only during training, while PPG serves as the primary modality available during both training and inference.

The Cardiovascular Health Monitoring

The experimental results for AF detection and blood pressure classification are summarized in Table 2. First, we observe that unimodal methods designed to address inter-subject variability in physiological signals, such as BPD [Su et al., 2022], achieve better cross-subject test performance compared to the vanilla supervised learning counterpart. Second, certain multimodal coordination methods, such as SimCLR [Chen et al., 2020], outperform unimodal counterparts, demonstrating the effectiveness of multimodal coordination methods when only unimodal data is available during deployment. In contrast, the two multimodal DG methods [Dong et al., 2023; Fan et al., 2024] show inferior performance, likely due to the strong dominance of ECG/ABP signals in their framework, which may hinder the models' generalization when only PPG signals are available at test time.

Overall, our proposed method consistently outperforms all baseline approaches across all three evaluation metrics on two cardiovascular health monitoring tasks. It achieves about 2.5% absolute improvement in Accuracy, Macro F1, and AUC ROC scores for the AF detection task compared to the second-best method. For the blood pressure classification task, it improves Accuracy by 1%, Macro F1 by 2%, and AUC ROC scores by 1.6%. Specifically, the proposed method achieves statistically significant improvements over the second-best method across all evaluation metrics.

The Sleep Stage Classification

Table 3 presents experimental results of the sleep stage classification task on the MESA subset. Consistent with the findings in the previous section, multimodal coordination methods, including SimCLR [Chen et al., 2020] and MERL [Liu et al., 2024], outperform unimodal approaches. The proposed framework, PhysioGMC, achieves the best performance among all evaluated methods. These results demonstrate the superiority of learning from multiple physiological signals in a collaborative manner during training, even when only unimodal wearable signals are available at deployment, offering more practical and comfortable options for continuous sleep monitoring.

Extension to Semi-supervised Learning Settings

As discussed in Section 4.2, the proposed framework can extend to semi-supervised learning settings where models are trained using both labeled data and unlabeled data. In this section, we evaluate the performance of PhysioGMC in the semi-supervised scenario by randomly masking labels at ratios of {0%, 40%, 70%}. We compare PhysioGMC against the unimodal supervised learning baseline trained only on the available labeled data. As shown in Table 4, PhysioGMC outperforms the baseline across all label-missing ratios, demonstrating its scalability in semi-supervised settings by leveraging unlabeled data through the self-supervised cross-modal contrastive loss in the training process.

5.3 Ablation Experiments

To further illustrate the contributions of each module in the proposed framework, we conduct ablation experiments evaluating the impact of each component of the proposed PhysioGMC framework on the MIMIC-AF and MIMIC-Hyper datasets. The results of ablation experiments, as shown in Table 5, demonstrate the effectiveness of both self-supervised and supervised cross-modal contrastive losses. The B2 variant, which incorporates the self-supervised contrastive loss only, achieves notable improvements in performance compared to A1 under unimodal testing scenarios, demonstrating the advantage of multimodal coordination paradigms that encourage learning modality-consistent features. The proposed framework, which additionally incorporates the supervised contrastive loss, further enhances cross-subject performance over B2 by explicitly enforcing the disentanglement of subject-agnostic information and subject-specific information in the learned multimodal label-decisive features.

We also evaluate the effectiveness of the proposed supervised contrastive loss when training with a single modality. Compared to the unimodal baseline A1, variant A2 achieves improved cross-subject test performance, suggesting its ability to learn label-relevant representations agnostic to subjects. However, the proposed framework with multimodal contrastive learning modules outperforms the unimodal variant, demonstrating the advantage of learning subject-agnostic, label-relevant features collaboratively across multiple modalities rather than from a single modality.

5.4 Feature Visualization Analysis

In this section, we utilize t-SNE [Maaten and Hinton, 2008] to visualize the PPG features, zp i,a , learned by the baseline unimodal model and PhysioGMC on the MIMIC-AF dataset. As shown in Fig. 4, the embeddings learned by PhysioGMC exhibit clearer, more distinguishable classification boundaries. For subjects without AF, the distribution of PPG embeddings is inherently more dispersed due to heterogeneous cardiovascular conditions. Nevertheless, the proposed framework yields a more compact distribution for the non-AF cohort than the unimodal baseline, demonstrating its ability to suppress subject-specific information in the extracted label-relevant features. While there remains some overlap between classes, reflecting the challenges brought by inter-subject variability of PPG signals, these results show improved feature separability for our framework. Further exploration may help achieve more distinct decision boundaries for unseen subjects.

(b) Proposed PhysioGMC

Figure 4: t-SNE visualization of PPG features (zp i,a ) on the MIMICAF dataset using (a) a unimodal EfficientNet baseline and (b) the proposed PhysioGMC framework.

6 Conclusion

We propose PhysioGMC, a generalizable multimodal coordination framework for physiological signals that leverages both clinical-grade and wearable modalities during training. The proposed framework learns label-relevant, subjectagnostic features collaboratively across multiple modalities by integrating proposed self-supervised and supervised crossmodal contrastive losses, leading to improved cross-subject performance for wearable modalities under unimodal deployment scenarios. Experiments on cardiovascular health monitoring and sleep stage classification tasks show that PhysioGMC consistently achieves superior cross-subject performance when only a single wearable physiological signal, PPG, is available during testing for these tasks. This demonstrates the potential of PhysioGMC to support continuous, accessible, and reliable health monitoring using only wearable sensing in real-world applications.

Acknowledgments

This work was supported in part by the U.S. National Science Foundation (NSF) grant 214291.