Bridge: A Cross-Modal Learning Framework for Unified Semantic Representation in Noisy Communication

1 Introduction

Traditional communication systems predominantly rely on the accurate transmission of raw bit streams. However, with the rapid advancement of artificial intelligence (AI) and intelligent devices, increasing attention has been directed toward the semantic value of information [Yang et al., 2023; Xia et al., 2024]. Unlike raw data, semantic information aligns more closely with human cognition and task-specific requirements, enabling more efficient and purposeful communication under stringent bandwidth constraints. Semantic communication has thus emerged as a transformative paradigm, prioritizing the conveyance of intended meaning over the faithful delivery of original data [Wang et al., 2023b]. This shift not only aligns with the intelligence-driven demands anticipated in future 6G networks but also paves the way toward communication systems endowed with cognitive and context-aware capabilities.

∗Corresponding author.

Current research efforts in semantic communication have primarily focused on unimodal data (e.g., text, audio, and video), emphasizing semantic extraction, compression, and transmission. For textual modalities, natural language processing techniques, including word embeddings and Transformer architectures, have enabled effective semantic representation and delivery [Miao et al., 2024]. In the audio domain, the integration of automatic speech recognition and semantic inference has improved the preservation of speaker intent [Chen et al., 2024]. For video-based communication, convolutional neural networks and temporal modeling approaches have been employed to capture and transmit core semantic content while minimizing redundancy [Hwang et al., 2024]. Despite the progress, unimodal semantic systems still face several critical challenges, including ambiguous semantic definitions, limited channel robustness [Cheng et al., 2024], and high system complexity, all of which hinder scalability and adaptability in dynamic real-world scenarios.

To overcome these limitations and enhance the depth and flexibility of semantic representation, research focus has increasingly shifted toward multimodal semantic communication [Lu et al., 2024]. By integrating heterogeneous data sources, multimodal systems offer a more comprehensive understanding of complex semantics [Liu et al., 2024]. Theoretical advances in this domain have introduced concepts such as multimodal semantic entropy and cross-modal mutual information, while early frameworks for joint semantic encoding have been proposed. Technologically, advances in multimodal Transformers, graph neural networks, and attention mechanisms have significantly enhanced the modeling of semantic relationships across diverse modalities [Cheng et al., 2025]. These developments show promising potential for applications in autonomous driving, intelligent healthcare, and immersive virtual environments [Tang et al., 2024]. Nevertheless, current multimodal semantic systems often face challenges such as inefficient cross-modal alignment, limited real-time processing capabilities [Nguyen et al., 2024], and the absence of unified evaluation standards, all of which present obstacles to large-scale deployment.

To address these challenges, previous studies have shown that emergent protocols optimized solely for utility often fail to produce semantically consistent mappings [Ben Zion et al., 2024], highlighting the need for a unified and reconstructionoriented approach to semantic learning. In response, this paper introduces a novel multimodal semantic framework that projects heterogeneous modalities into a unified latent feature space and employs alignment mechanisms to ensure coherent semantic representation. This design significantly improves both the robustness of communication and semantic fidelity in complex and dynamic environments. Compared to existing methods, the proposed framework achieves notable improvements in semantic alignment accuracy, system adaptability, and resilience to channel disturbances [Zhao et al., 2024].

The contributions of this paper are summarized as follows.

1. We propose a cross-modal learning approach that effectively bridges the gap between video, audio, and text by learning a unified semantic representation. Our method disentangles modality-specific and modality-invariant features, enabling precise processing within each modality while ensuring robust generalization across modalities.
2. We introduce a novel cross-modal alignment mechanism that enhances semantic fidelity by aligning features from different modalities in a shared latent space. This method is designed to maintain consistent semantic mapping across modalities, even in the presence of noise and fading, addressing the challenges of communication channels.
3. Extensive experiments on various multimodal benchmarks demonstrate that BRIDGE outperforms existing methods, particularly in low-SNR conditions. Our results show that BRIDGE preserves semantic relationships across modalities while maintaining robust performance, improving image reconstruction (PSNR) by approximately 16.3% in low-SNR regimes, validating its effectiveness in applications.

2 RelatedWork

Unimodal Semantic Communication. Existing research on semantic communication has primarily focused on unimodal data (e.g., text, speech/audio, and vision), aiming to transmit task-relevant semantics rather than raw bitstreams to improve communication efficiency under bandwidth constraints and channel impairments [Shao et al., 2022]. With the progress of deep learning, end-to-end semantic transceivers for text have become increasingly mature [Xie et al., 2021]. For speech/audio scenarios, task-driven semantic transmission has been explored by integrating recognition and synthesis objectives, enabling robust intent-preserving delivery over noisy channels [Weng et al., 2023]. For temporal signals such as video, semantic modeling often leverages spatiotemporal representation learning to capture both spatial cues and temporal dependencies for more faithful semantic extraction [Akbari et al., 2021]. In the visual domain, recent advances further incorporate stronger generative priors

(e.g., diffusion-based decoding) to enhance perceptual quality and robustness, especially in low-SNR regimes [Peng et al., 2025]. Despite these advances, unimodal semantic systems still face persistent challenges, including the ambiguity between "semantics" and task objectives, limited robustness under channel mismatch, and non-trivial system complexity for practical deployment [Xie et al., 2023].

Multimodal Semantic Communication. To overcome the limitations of unimodal systems and improve semantic completeness, research attention has gradually shifted to multimodal semantic communication, where heterogeneous sources are jointly exploited for more comprehensive understanding and more reliable downstream performance [Zhang et al., 2024; Xie et al., 2022]. Unlike unimodal settings, multimodal semantic communication must not only extract modality-specific features but also model inter-modal relationships and complementary cues to achieve context-aware and synergy-enhanced semantics . Methodologically, crossmodal alignment and fusion are often supported by contrastive and unified representation learning paradigms, exemplified by vision-language pretraining and mixture-of-experts style modeling that map heterogeneous inputs into shared semantic spaces [Radford et al., 2021; Bao et al., 2022]. However, practical multimodal semantic communication remains challenging due to the pronounced modality gap: heterogeneous modalities differ in distribution, granularity, and noise sensitivity, which can degrade alignment quality, and channel noise/fading further amplifies inconsistency at the receiver and causes unstable task performance [Xie et al., 2022].

Unified SemanticRepresentation. Acentral goal of crossmodal learning is to build a unified semantic representation by projecting heterogeneous modalities into a shared latent space and enforcing alignment to preserve semantic consistency [Li et al., 2021]. Recently, scalable pretraining frameworks have strengthened this direction by combining shared semantic spaces with modality-aware architectures (e.g., modality-specific branches or expert components), leading to more expressive and transferable crossmodal representations [Wang et al., 2023a; Li et al., 2023]. Nevertheless, most alignment strategies are developed under relatively stable data assumptions. When transferred to wireless semantic communication, channel perturbations can reshape the transmitted feature distribution and introduce uncertainty, which may induce semantic drift and crossmodal mismatch under noise and fading [Xie et al., 2023; Zhang et al., 2026]. Moreover, under joint constraints of bandwidth limitation, channel variation, and deployment requirements (complexity/latency), achieving accurate alignment, robust generalization, and practical efficiency simultaneously remains an open challenge [Shao et al., 2022]. This is particularly difficult in real-world applications, where environmental factors constantly change and affect the performance of cross-modal models.

Motivation. Motivated by these observations, key questions arises for next-generation multimodal communication: how can we learn a unified semantic representation that is (i) consistent across heterogeneous modalities, (ii) robust to noise, fading, and channel mismatch, and (iii) efficient enough for practical deployment under bandwidth and latency constraints. Addressing questions is crucial for enabling reliable multimodal semantic delivery and downstream performance in dynamic real-world environments.

3 Methodology

In this section, we provide a detailed description of the proposed multimodal semantic communication framework, Bridge. The core objective of Bridge is to enable collaborative expression and robust transmission of multimodal data within a unified semantic space through feature disentanglement and cross-modal alignment mechanisms. To facilitate a clearer understanding of the overall workflow and modular structure of the framework, Figure 1 illustrates the architecture of Bridge. Specifically, inputs from text, audio, and video modalities are first processed by modality-specific feature extractors, which generate both modality-specific and modalityinvariant representations. The modality-invariant features are subsequently quantized and transmitted to the receiver side. A two-stage decoder is employed for semantic reconstruction: in the first stage, lightweight modality-specific decoders are used to reconstruct preliminary modality signals; in the second stage, large-scale foundation models are leveraged to verify and enhance semantic consistency.

Feature Extraction Module. In multimodal semantic communication, different modalities simultaneously carry modality-specific characteristics and invariant semantic content. To effectively extract and separate these aspects, we design a dual-branch encoder architecture. Each input modality is processed through two parallel pathways: one branch is responsible for extracting modality-specific features, while the other captures modality-invariant representations. These two branches are trained jointly but kept structurally independent to ensure specialization and avoid semantic interference.

This branch is designed to preserve the unique and irreplaceable characteristics intrinsic to each modality. For a given input xm from modality m ∈ {a, b, c}, we first extract initial features using the modality-specific encoder ϕm(·). These features are then projected and compressed using a projection head MLPmspec . To improve robustness, Gaussian noise is added as a regularization term. The resulting modalityspecific feature is defined as

This representation is not quantized and is used in downstream tasks where fidelity and integrity are important.

To facilitate semantic alignment across modalities, we introduce a parallel branch for extracting modality-invariant features. This branch is built upon a shared semantic encoder that projects all modalities into a common latent space.

Specifically, for each modality m ∈ {a, b, c}, the input xm is first encoded by the same low-level modality-specific encoder ϕm(·). The output is then passed to a shared encoder gshared(·) to produce the semantic representation as follows.

These modality-invariant features are subsequently used in the specific-invariant dependency minimization and the modality-aware cross-modal semantics for optimizing feature separation and cross-modal semantic consistency.

Specific-Invariant Dependency Minimization (SDMI). To ensure effective separation between modality-specific and modality-invariant features, we enhance the SDMI module by introducing auxiliary projection heads (distinct fromMLPmspec used in feature extraction), which are solely used for robust mutual information estimation.

To facilitate mutual information estimation while preserving numerical stability, we introduce two auxiliary projection heads ψm and ψinv, which map zm and zinv into lowdimensional semantic spaces only for mutual information estimation. This design is based on the invariance property of mutual information, i.e. I(zm; zinv ) = I(ψm(zm); ψinv(zinv )) under deterministic mappings.

We enhance the contrastive estimation framework by expanding the Mutual Information Neural Estimator (MINE), which approximates mutual information via the Donsker-Varadhan representation as follows.

where Tθ is a neural critic operating on the projected features. It provides a tighter and smoother gradient signal and eliminates the need for explicit negative sampling.

These enhancements allow SDMI to more effectively suppress residual redundancy between the shared and specific latent factors, leading to improved disentanglement and more robust cross-modal generalization. across modalities.

Modality-Aware Cross-modal Semantics (MACS). To align modality-invariant features in a shared semantic space, we introduce MACS, a pairwise-gated contrastive module. MACS forms anchor-candidate pairs across modalities, maps them with two asymmetric projection heads ϕa and ϕb, and measures compatibility through bilinear similarity. A learnable pairwise gate then scales each pair's logit inside a negative InfoNCE objective to emphasize reliable matches and suppress noisy or weak alignments as follows.

where the pairwise similarity is

The gate for each pair (t, j) is produced from the concatenated projections and squashed by a sigmoid as follows.

This design is equivalent to using an adaptive temperature with gt,j = 1/τt,j: low cross-modal consistency increases τt,j , whereas high consistency decreases τt,j , thereby enforcing explicit cross-modal structural consistency.

Vector Quantization. After the alignment phase, the modality-invariant features are discretized via vector quantization (VQ), converting continuous latent semantics into discrete code indices. This step improves compactness and maps cross-modal semantics to a unified discrete space for downstream reconstruction and reasoning.

Let zm ∈RT×D be the invariant representation of modality m. We quantize each time step by nearest-neighbor lookup in a shared codebook {ej}Kj=1 ⊂ RD as follows.

Training Mechanism. We train the discrete bottleneck with the straight-through estimator (STE). In the backward pass the quantizer is treated as identity by using a stopgradient operator sg[·] as follows.

The VQ loss combines a codebook term and a commitment term as follows.

where β > 0 (we use β = 0.25 unless stated otherwise). The first term updates code vectors toward assigned latents, and the second encourages latents to stay close to selected codes.

Cross-modal Usage Monitoring and Reset. To mitigate codebook collapse (few codes overused, many inactive), we adopt a windowed cross-modal usage rule. Let M be the set of modalities and u(m) i (t′ )∈{0, 1} indicate whether code ei is used by modality m at step t′ . Within the most recent Nreset steps, we count the number of distinct modalities that have used ei as follows.

If Ci < 2 (i.e., the code is not shared by at least two modalities within the window), we re-initialize it as follows.

This strategy periodically resets inactive or modalityexclusive codes, improving codebook utilization and promoting robust, shared semantic encoding across modalities.

Semantic Decoding with Foundation Models. On the receiver side, a multimodal semantic decoding and reconstruction module is designed to recover the underlying semantic representations of audio, video, and text from the compressed and quantized vectors transmitted over the channel. Since these vectors are modality-invariant and aligned in a shared latent space, decoding must focus not only on surface-level signal restoration, but also on preserving the high-level semantic structures originally embedded.

To achieve this, the system employs a two-stage decoding process. First, modality-specific decoders are used to reconstruct intermediate modality data-such as audio waveforms, video frames, and text tokens-from quantized latent representations. These decoders are lightweight and trained jointly with the encoder to maintain low latency and acceptable fidelity under constrained transmission conditions.

We define the reconstruction objective as a weighted sum of modality-specific reconstruction losses as follows.

where ˆxa , ˆxv , ˆxt are reconstructed audio/video/text signals, xa , xv , xt denote their original counterparts. We use mean squared error (MSE) for continuous modalities (audio and video) and cross-entropy (CE) for discrete text prediction.

In the second stage, a foundation model alignment mechanism is introduced to perform semantic verification and enhancement. Each reconstructed modality-specific signal is passed through a pre-trained foundation model that maps it into a high-dimensional semantic embedding space.

where f (a) sem, f (v) sem, and f (t) sem denote CLAP [Elizalde et al., 2023], CLIP [Radford et al., 2021], and BERT [Devlin et al., 2019], respectively, which project the (reconstructed) signals into a shared d-dimensional semantic embedding space. This mapping enables evaluation of semantic consistency and quality using learned representations.

Specifically, for the audio modality, CLAP is employed to project the waveform into an audio-language embedding space, ensuring semantic alignment with the intended meaning. For the video modality, CLIP analyzes reconstructed frames using a dual-encoder architecture that bridges visual and textual semantics. For text, BERT encodes token sequences into contextualized embeddings, enabling assessment of both lexical accuracy and semantic coherence.

This foundation model-enhanced decoding mechanism enables semantic validation, cross-modal embedding consistency, and robustness under transmission degradation.

Overall Objective. To enable effective end-to-end optimization of BRIDGE, we aggregate the losses from each component into a single objective as follows.

Here, LSDMI minimizes the mutual information between modality-specific and modality-invariant branches via a DV/MINE critic and a min-max schedule; LMACS is a negative InfoNCE objective with pairwise gating gt,j (interpretable as an adaptive temperature) to enforce cross-modal semantic alignment; Lrecon is the sum of modality-wise reconstruction losses (audio, video, and text); and LVQ is the vector-quantization loss combining codebook and commitment terms under a straight-through estimator. The coefficients α1-4 are chosen on the validation set to balance gradient magnitudes and overall performance.

4 Experiments

In this section, we describe the experimental setup, including the datasets and tasks used. To comprehensively evaluate the robustness, semantic expressiveness, and downstream adaptability of the proposed semantic communication framework, we conduct experiments using the VGGSound [Chen et al., 2020], COCO2017 [Lin et al., 2014], CMU-MOSEI [Zadeh et al., 2018], and MM-IMDb [Ovalle et al., 2017] datasets.

Audio and Video Reconstruction on VGGSound. Figures 2 summarizes reconstruction quality under AWGN across SNRs from -15 to 15dB, comparing BRIDGE with ImageBind [Girdhar et al., 2023] and the general-purpose Uni-Perceiver v2 [Zhu et al., 2022]. Overall, BRIDGE exhibits consistently stronger robustness across the entire SNR spectrum: it achieves higher semantic similarity for both audio and video, with the most visible gains in low-to-medium SNR regimes, and quickly approaches near-perfect similarity once the channel becomes moderate. Meanwhile, BRIDGE attains markedly better signal/perceptual quality, showing much lower audio reconstruction error and higher audio SISDR as SNR increases, as well as substantially higher video PSNR and SSIM, indicating improved pixel-level fidelity and structural preservation of frames. These suggest that disentangled invariant representation together with modality-aware cross-modal alignment enables BRIDGE to simultaneously preserve high-level semantics and low-level reconstruction details under noise, outperforming both unified embedding baselines and a generic multimodal backbone.

Image Reconstruction Quantitative Comparison. On COCO2017, Table 1 reports PSNR and LPIPS under AWGN

and Rayleigh channels at SNR ∈ {0, 5, 10, 15} dB. Compared with representative JSCC baselines, including the semantics-guided diffusion approach SGD-JSCC [Zhang et al., 2026], BRIDGE consistently achieves the best reconstruction quality across all settings, yielding higher PSNR and lower LPIPS. The gain is most pronounced in low-SNR regimes, where BRIDGE better preserves structural details and perceptual fidelity under severe noise and fading. As SNR increases, BRIDGE continues to improve and maintains stable advantages, demonstrating robust performance under both noise and fading.

Multimodal Sentiment Classification Comparison. To further evaluate BRIDGE on multimodal task under different SNRs. As reported in Table 2, BRIDGE achieves the best classification accuracy across the entire SNR range, showing clear advantages especially in low-SNR regimes and maintaining stable gains as channel conditions improve, which demonstrates stronger robustness for sentiment-related semantic preservation under noise.

Multimodal Genre Classification Comparison. Figure 3 compares BRIDGE with MFSC, MFMSC, MMSC [Zhu et al., 2025], U-DeepSC, and T-DeepSC under different SNRs in AWGN and Rayleigh channels. Overall, BRIDGE consistently achieves the best performance across the SNR range in both settings, showing clear advantages especially as channel conditions improve. These results indicate that BRIDGE provides stronger robustness and more reliable semantic preservation under both additive noise and fading.

Image Reconstruction Trend Analysis. Figure 4 presents the PSNR and SSIM trends on the COCO2017 dataset under both AWGN and Rayleigh channels. Compared with the UIDSC approach [Ye et al., 2025], the heatmap results further highlight the across-the-board robustness of Bridge: it maintains stronger overall performance over the entire SNR spectrum and exhibits better generalization under varying and mismatched channel conditions.

Ablation Study. To quantify the contribution of the two core components in Bridge-SDMI for shared/specific disentanglement and semantic purification, and MACS for crossmodal contrastive semantic alignment-we conduct a controlled ablation on VGGSound under four representative SNRs (-10, -5, 5, and 10 dB). Figure 5 visualizes the results with six metrics (Audio Sim, Audio MSE, Audio SI-SDR, Video Sim, Video PSNR, and Video SSIM), where larger values are better except for MSE. In Figure 5, the farther the value is from the center, the better the performance.

Across all SNRs, the full Bridge yields the outermost radar profile, indicating the best overall multimodal reconstruction quality. Removing SDMI causes a uniform degradation in both modalities, especially in low SNR regimes: at -10 dB, Audio Sim drops from 0.223 to 0.198, SI-SDR decreases from -11.09 to -13.06 dB, and Video PSNR falls from 5.58 to 4.53 dB, suggesting that SDMI is crucial for suppressing modality-specific noise leakage and stabilizing semantic codes under severe channel corruption. Removing MACS leads to an even larger decline in cross-modal consistency, reflected by simultaneous drops in Audio/Video Sim and perceptual fidelity; e.g., at -10 dB the Video Sim decreases from 0.207 to 0.133 and SSIM from 0.213 to 0.124, highlighting MACS as the main driver for maintaining aligned semantics across audio-video streams. As SNR increases, all variants improve, but Bridge preserves a clear margin, particularly on signal-level quality: at 10 dB, Bridge reaches 15.23 dB SISDR and 22.50 dB PSNR, while the two ablations remain lower (e.g., 14.29/21.63 for w/o SDMI and 10.45/19.69 for w/o MACS). The baseline without these components stays inside the radar across all SNRs, confirming that Bridge's robustness stems from the effect of SDMI-based disentanglement and MACS-based cross-modal alignment.

5 Conclusion

This paper presented Bridge, a novel cross-modal learning framework that effectively bridges modal gaps in semantic communication through unified representation learning. Our key innovations include: Disentangled Feature Learning for Unified Cross-Modal Representation, which disentangles modality-specific and modality-invariant features to improve both intra-modal precision and inter-modal generalization; and Improved Cross-Modal Alignment under Noisy Channel Conditions, which aligns heterogeneous features in a shared latent space to preserve semantic consistency under noise and fading. Extensive experiments on multiple multimodal benchmarks further demonstrate that BRIDGE achieves strong robustness in low-SNR environments and preserves cross-modal semantic with stable task performance.

Acknowledgments

This work is supported by the Natural Science Foundation of Fujian Province (No. 2024J09032, No. 2025J01379, No. 2025H0043, No. 2025J02019), the Joint Funds for the Innovation of Science and Technology of Fujian Province (No. 2024Y9491).