scLLM-DSC: LLM-Knowledge Enhanced Cross-Modal Deep Structural Clustering for Single-Cell RNA Sequencing

1 Introduction

Single-cell RNA sequencing (scRNA-seq) has revolutionized the characterization of tissue heterogeneity by providing highresolution cellular atlases [Svensson et al., 2018]. As a pivotal analytical step, cell clustering aims to define functional identities based on expression similarity, serving as the cornerstone for downstream biological discovery [Kiselev et al., 2019].

In retrospect, the evolution of clustering algorithms has been a pursuit of optimal numerical representations. From early statistical linear reductions (e.g., PCA, t-SNE [Maaten and Hinton, 2008]) to deep autoencoder-based non-linear embeddings (e.g., scDeepCluster [Tian et al., 2019], scNAME [Wan et al., 2022]), and further to structure-constrained learning (e.g., scGNN [Wang et al., 2021] and scCDCG [Xu et al., 2024]), existing methods have achieved maturity in mining statistical distributions and topological structures. However, these methods suffer from a fundamental limitation: semantic agnosticism [Cui et al., 2024; Theodoris et al., 2023]. By reducing genes to abstract numerical indices, existing frameworks prioritize expression patterns over functional logic. Consequently, while adept at identifying distributional shifts, they lack the semantic grounding required for biological reasoning and interpretability [Xu et al., 2025d].

* Corresponding authors.

To address this limitation, the field is witnessing a paradigm shift toward Foundation Models (FMs) [Zhang et al., 2025]. Inspired by LLMs in NLP, these approaches analogize genes as "tokens" and cells as "sentences," pre-training on massive atlases to learn universal representations. Pioneering works like scBERT [Yang et al., 2022] and Geneformer [Theodoris et al., 2023] adopted the BERT paradigm, capturing genome-wide context via masked modeling. Subsequently, generative models such as scGPT [Cui et al., 2024] and scFoundation [Hao et al., 2024] scaled up parameters to reconstruct complex expression patterns across species. More recently, researchers have sought to enhance interpretability by integrating biological priors, such as gene regulatory networks in GeneCompass [Yang et al., 2024] or leveraging textual embeddings in GenePT [Chen and Zou, 2024].

Despite their potential, directly adapting these generalpurpose models to the specific downstream task of cell clustering presents structural contradictions [Kedzierska et al., 2025; Li et al., 2025]: (1) Objective Mismatch between Pretraining and Clustering: Most models (e.g., scGPT) rely on "generative" pre-training focused on reconstructing global statistics rather than distinguishing local subpopulation boundaries. When applied to clustering (a discriminative task), this lack of boundary constraints often leads to "over-smoothing" representations, failing to separate subtle transitional states. (2) Numerical representations lack biological semantic support: While treating genes as tokens, existing methods essentially capture numerical co-occurrence rather than explicit functional attributes (e.g., transcription factor regulation). Prioritizing probabilistic associations over causal logic makes them prone to hallucinations, thereby generating biologically implausible patterns driven by noise. (3) Missing CrossModal Alignment: There is a lack of effective mechanisms to align "external biological knowledge" with "internal transcriptomic features" in the latent space. This modal disconnection results in clustering outputs that remain mathematical labels rather than biologically attributable entities [Dip et al., 2025].

To address these impediments, we propose scLLM-DSC, an LLM-Knowledge Enhanced Cross-Modal Deep Structural Clustering framework for single-cell RNA Sequencing. Diverging from the purely generative paradigm of general foundation models, scLLM-DSC is formulated as a task-specific knowledge-integrated framework anchored in three core pillars: Knowledge-Driven Priors, Structure-Aware Topology, and Cross-Modal Semantic Alignment. The core logic is to utilize LLMs not as data generators but as semantic mappers, explicitly integrating biological semantics from knowledge bases (e.g., NCBI) into the clustering process and enforcing alignment with numerical features via cross-modal contrastive learning. scLLM-DSC comprises three coupled modules: (1) Dual-Path Cell Semantic Encoding (Knowledgedriven): It incorporates external priors by encoding NCBI gene summaries and using a Cell2Sentence strategy to capture dynamic regulation contexts. (2) Structure-Aware Cell Feature Encoding(Structure-aware): Leveraging the scCDCG backbone, it extracts high-order topological information to ensure the fidelity of the native data manifold. (3) CrossModal Alignment and Fusion (Semantic-alignment): Utilizing bidirectional InfoNCE loss and variance regularization, this module bridges the cognitive gap between "data-driven" structural signals and "knowledge-driven" semantic signals. Extensive benchmarks demonstrate that scLLM-DSC significantly outperforms 11 mainstream baselines in both accuracy and efficiency, while uniquely endowing clustering results with explicit biological attribution.

Our contributions are summarized as follows:

• A Knowledge-Data Dual-Driven Paradigm: We transcend the limitations of traditional numerical indexing by systematically incorporating NCBI functional priors. This establishes a novel paradigm that bridges the gap between 'structure-aware' deep learning and semanticaware' biological reasoning.
• Task-Specific Synergistic Optimization: To overcome the objective mismatch of FMs for clustering, we devise a specialized optimization objective. By integrating semantic alignment with discriminative structural constraints, we ensure the model focuses on defining precise cell subpopulation boundaries rather than global reconstruction.
• Unified Interpretable Representation: We leverage cross-modal alignment to construct a unified latent space that preserves both topological fidelity and biological meaning, enabling interpretable mechanism discovery.

2 RelatedWork

Conventional scRNA-seq Clustering. Early pipelines like Seurat [Butler et al., 2018] and Scanpy [Wolf et al., 2018] rely on linear reduction and community detection. To capture non-linear manifolds and data sparsity, deep generative models such as scVI [Lopez et al., 2018] and DCA [Eraslan et al., 2019] utilize ZINB-integrated autoencoders for explicit count modeling. Subsequent frameworks, including scDeepCluster [Tian et al., 2019] and scDCC [Tian et al., 2021], introduced joint optimization of reconstruction and clusteringconstrained objectives. To incorporate intercellular topological information, initial frameworks like scGNN [Wang et al., 2021] and scDSC [Gan et al., 2022] utilize graph convolutional networks for neighborhood aggregation, while scSiameseClu [Xu et al., 2025a] employs contrastive learning to bolster representation robustness. To enhance scalability and circumvent GCN-induced over-smoothing, structural models based on deep graph-cut objectives have emerged. scCDCG [Xu et al., 2024] and its extension scSGC [Xu et al., 2025b] utilize deep spectral clustering to efficiently capture global topological constraints. Despite these advancements, these numerical-driven methods often lack explicit biological semantics, limiting their interpretability in complex tissues. Biological Foundation Models. The focus has recently shifted toward Transformer-based foundation models that learn universal biological representations from large-scale cell atlases. scBERT [Yang et al., 2022] and Geneformer [Theodoris et al., 2023] pioneered this paradigm by employing masked gene modeling to capture complex genomic contexts. To enhance generalization, scGPT [Cui et al., 2024] and scFoundation [Hao et al., 2024] further scaled parameter capacities for multi-species generative embeddings. Beyond sequence-only modeling, knowledge-enhanced frameworks such as GeneCompass [Yang et al., 2024] and GenePT [Chen and Zou, 2024] integrate gene regulatory networks and textual semantics, respectively, to infuse prior biological constraints. Recently, scMamba [Yuan et al., 2025] was introduced to achieve linear computational complexity for ultra-long genomic sequences. Despite their powerful representation, these foundation models are not inherently optimized for partitioning tasks, suffering from a objective mismatch between general pre-training and downstream structural clustering.

3 Methodology

3.1 ProblemFormulation

The core challenge addressed in this study is KnowledgeEnhanced Structural Clustering. Let X ∈ RN×D be the single-cell gene expression matrix, where N denotes the number of cells and D is the number of genes. Unlike traditional data-driven methods, we incorporate an external gene semantic knowledge base, denoted as G, derived from LLMs. Our objective is to learn a mapping function f : (X,G) → Z that projects cells into a unified latent space Z by aligning transcriptomic structural features with biological semantic information. Ultimately, we aim to partition Z into O cell populations, yielding the clustering assignment C = {c1, c2, . . . , cN}, where ci ∈ {1, . . . , O} denotes the cluster label of the i-th cell.

3.2 Framework Overview

The scLLM-DSC framework, illustrated in Fig. 2, establishes a novel paradigm for LLM-Knowledge enhanced cross-modal deep structural clustering by synergizing large-scale biological priors with data-driven representation learning. To integrate biological semantic knowledge, we first retrieve structured biological annotations from NCBI and encode them into genelevel textual representations using a frozen large language model. Subsequently, we construct a unified cell embedding through three modules: (1) Semantic View: creating complementary cell-level textual embeddings by aggregating gene semantics and serializing cell profiles; (2) Structural View: capturing the intrinsic data manifold via a graph-cut-guided deep structural encoder; and (3) Cross-Modal Contrastive Fusion: performing alignment between LLM-encoded biological semantics (Text) and expression-derived structural features (Feature), yielding a biologically grounded latent space optimized for the downstream clustering task.

3.3 Gene Semantic Encoding

To explicitly incorporate biological priors, we leverage a pretrained LLM as a universal knowledge encoder, as illustrated in Fig. 1. Specifically, we systematically collected comprehensive metadata covering genome-wide genes from the NCBI database [Li et al., 2024]. For the j-th gene (j ∈ {1, . . . ,M}), this heterogeneous information (e.g., symbol, functional summary) is serialized into a structured prompt Tj . We then extract the dense vector representation via the LLM encoder:

Consequently, we construct a global gene semantic matrix

which projects discrete gene entities into a continuous, machine-interpretable semantic manifold.

3.4 Dual-Path Cell Semantic Encoding

To bridge the gap between gene-level knowledge and cell-level identity, we design a dual-path encoding strategy. Crucially, both paths operate on the most informative genes to filter noise and ensure computational efficiency. For each cell, we strictly select the top-K genes ranked by expression intensity (experimentally set to K = 2048). Let ˜X ∈ RN×K denote this filtered and sorted expression matrix, where each row contains the expression values of the top-K genes for a cell. Abundance-Weighted Semantic Aggregation. This branch explicitly utilizes the expression magnitudes in ˜X as importance weights. Let ˜G ∈ RK×d1 be the subset of LLM-derived gene embeddings corresponding to the genes in ˜X. The first cell-level embedding Z(1) is obtained by aggregating these gene semantics weighted by their expression values:

which projects the cell's gene expression abundance directly into the semantic manifold defined by G. Sequence-Based Contextual Modeling. To capture nonlinear dependencies and gene co-occurrence patterns, we adopt a serialization approach inspired by Cell2Sentence [Levine et al., 2024]. We adopt a serialization approach where the row vector ˜xi from ˜X is treated as a sentence. Specifically, the names of the top-K genes (ordered by their rank in ˜X)

are tokenized into a sequence Si. These sequences are then encoded by the shared LLM encoder fLLM:

Unlike Z(1) , this representation leverages the LLM's attention mechanism to perceive the underlying regulatory logic of gene expressions preserved in the sorted gene list.

Dual-Path Semantic Fusion. We integrate the abundancebased and context-based representations to yield the final unified cell semantic embedding Ztext :

where ω ∈ [0, 1] is a balancing coefficient. This fused representation Ztext ∈ RN×d1 serves as the semantic anchor for the subsequent structure-aware alignment.

3.5 Structure-Aware Cell Feature Encoding

To capture the intrinsic topological manifold of the cell population, we utilize the deep graph clustering framework scCDCG [Xu et al., 2024] as our structural backbone. Given input X, the backbone functions as a non-linear mapping encoder fstruc(·) to extract the cell feature embedding:

This encoding process is governed by three complementary unsupervised objectives:

• Topology Preservation (LNCut): A differentiable Normalized Cut loss that captures high-order structural information via long-range dependencies, effectively preserving the global manifold while circumventing the oversmoothing limitations of traditional GCNs.
• Expression Fidelity (LMSE): An autoencoder-based reconstruction loss that guarantees the latent embeddings retain essential gene expression information.
• Cluster Refinement (LKL): A KL-divergence objective that aligns soft assignments with a target distribution optimized via Optimal Transport (OT). This mechanism enforces a global balance constraint to prevent cluster collapsing and stabilize the partitioning manifold.

The resulting Zfeat encapsulates the data-driven structural priors, serving as the topological anchor for the subsequent cross-modal alignment.

3.6 Cross-Modal Alignment and Fusion

To bridge the heterogeneity between biological semantics and topological features, we align the semantic embedding Ztext and the structural embedding Zfeat into a unified latent space via contrastive learning.

Projection and Similarity. We first map the modality-specific embeddings to a shared d-dimensional space using two separate non-linear projection heads (two-layer MLPs with ReLU), denoted as fϕ(·) and gψ(·):

where ˆZ ∈ RN×d . We then compute the cross-modal cosine similarity matrix S ∈ RN×N . By normalizing the projected embeddings to the unit hypersphere (i.e., ∥ˆz∥ = 1), the similarity is efficiently calculated via scaled dot product:

where τ is a temperature parameter.

Bidirectional Alignment Objective. To enforce semantic consistency, we minimize the symmetric InfoNCE loss. For a mini-batch of N cells, considering the paired representations (i, i) as positives and all other pairs as negatives, the loss is formulated as:

The first term aligns the textual view to the structural view, while the second aligns the structural view to the textual view. Variance Regularization. To prevent the projection heads frommapping diverse cells to a trivial point (i.e., dimensional collapse), we impose a hinge loss on the standard deviation of each feature dimension:

where V:,k ∈ RN denotes the column vector corresponding to the k-th dimension across the batch, and ϵ = 10-4 is a constant for numerical stability.

The overall loss for the cross-modal alignment module is formulated as:

where λ is a hyperparameter.

3.7 Optimization and Inference

Joint Training. The framework is optimized end-to-end via a weighted composite objective:

where α, β, γ, and δ are hyperparameters balancing crossmodal semantic alignment, global topology preservation, reconstruction fidelity, and cluster refinement, respectively.

Inference. Upon convergence, we obtain the unified embeddings by averaging the aligned semantic and structural views:

The final clustering assignments C are then generated by applying K-Means (or Leiden) directly to Zcluster .

4 Experiments

4.1 Experimental Setup

Dataset. We utilized a subset of datasets provided by scCluBench [Xu et al., 2025c], consisting of 6 real scRNA-seq datasets from human and mouse tissues (Table 1). All data were used as provided, without further preprocessing.

Baseline Methods. We compare scLLM-DSC against 11 baselines across three categories: (1) Deep Non-linear Embedding Models: These models utilize deep autoencoders to learn latent representations, including scDeepCluster [Tian et al., 2019], scziDesk [Chen et al., 2020], and masked autoencoder variants scNAME [Wan et al., 2022] and scMAE [Fang et al., 2024]. (2) Deep Structural Clustering Models: These methods explicitly incorporate cellular topological information via graph-based architectures, comprising scGNN [Wang et al., 2021], scDSC [Gan et al., 2022], the contrastive scSiameseClu [Xu et al., 2025a], and the cut-informed scCDCG [Xu et al., 2024]. (3) Foundation Models: These large-scale pre-trained frameworks leverage expansive atlases for downstream transfer, represented by scGPT [Cui et al., 2024], GeneFormer [Theodoris et al., 2023], and the knowledge-informed GeneCompass [Yang et al., 2024]. All methods were implemented using their publicly available Python packages, with hyperparameters tuned for optimal performance.

Implementation Details. Implemented in Python 3.12 (PyTorch ≥2.9.1, CUDA 13.0), scLLM-DSC follows a twostage protocol: autoencoder pre-training (joint NCut/MSE) followed by contrastive clustering. We generate gene embeddings from NCBI metadata using OpenAI's text-embedding3-small. Cell embeddings are extracted from NCBI metadata via OpenAI's text-embedding-3-small. Cell embeddings aggregate the top-2048 highly expressed genes (k = 2048) weighted by expression vi: ecell = ∑ki=1 vi · egenei . The architecture features a [256, 16] autoencoder producing 16dimensional latent vectors. A 128-dimensional projection head aligns modalities via bi-directional InfoNCE loss (τ = 0.1) with variance regularization (λ = 10-2 ). Training spans 200 epochs/stage using Adam, with hyperparameters optimized via Optuna. Clusters are initialized by KMeans and refined via Sinkhorn (λ = 5, 1k iters). We report the mean of 5 independent runs on an NVIDIA A800-80GB GPU. All datasets and code for scLLM-DSC are available at the link: https://github.com/XPgogogo/scLLM-DSC.

Evaluation Metrics. We assess the clustering quality by measuring the congruence between predicted partitions and biological ground truth using three benchmark indices: Clustering Accuracy (ACC), Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI). While ACC and NMI range from 0 to 1, ARI can be negative if clustering performs worse than random assignment. For all metrics, higher values indicate superior clustering fidelity.

4.2 Overall Performance

Qualitative Analysis. (1)Comparison with Deep Clustering Baselines. Tab. 2 shows that scLLM-DSC achieves SOTA performance, outperforming all baselines in both average metrics (ACC: 88.80%, NMI: 85.35%, ARI: 83.04%) and overall ranking. The superiority stems from its ability to synergize LLM-derived semantic anchors with global structural manifolds. Specifically, the performance margin over deep non-linear embedding models underscores the necessity of integrating both textual semantics and topological structures. Furthermore, compared to deep structural clustering models (e.g., scCDCG), scLLM-DSC yields higher clustering fidelity, validating that LLM-based priors provide more discriminative guidance for structural alignment than raw numerical features. This robustness is further exemplified on the Muris Brain dataset (#Cell=13,417, #Cluster=2). While many baselines (e.g., scziDesk) achieve misleadingly high ACC by predicting majority classes, their near-zero NMI and ARI indicate a collapse into trivial partitions. Such failure occurs because raw numerical features often lack sufficient margin to define clear boundaries in high-volume but low-cluster scenarios, causing structural models to suffer from oscillation or over-smoothing. scLLM-DSC overcomes this by injecting LLM-based semantic anchors, which provide the contrastive signals required to maintain cluster separation and prevent manifold collapse. (2) Comparison with Biological Foundation Models. Evaluation against large-scale foundation models (Tab. 3) reveals that scLLM-DSC provides more robust partitioning across most datasets. Unlike general-purpose models, scLLM-DSC is specialized for clustering tasks, effectively capturing discrete cell-type boundaries. While Geneformer and GeneCompass show localized peaks on Sonya Liver, likely due to pre-training data overlap, scLLM-DSC demonstrates superior generalization elsewhere. This strongly underscores that for effectively resolving highly complex cellular heterogeneity, dedicated structural objectives specifically tailored for clustering are indispensable compared to universal embeddings.

Quantitative Analysis. As depicted in Fig. 3, we project the latent representations of scLLM-DSC and four representative baselines into a 2D space. The visualization results demonstrate that scLLM-DSC yields substantial advantages in both boundary clarity and intra-cluster compactness. Regarding the cluster boundaries, baselines such as scDeepCluster and scCDCG exhibit discernible "bridge" structures, resulting in marginal overlaps between distinct cell populations. Conversely, scLLM-DSC effectively eliminates such structural ambiguity; by leveraging LLM-based semantic priors to provide robust discriminative signals, it defines sharp and clean biological boundaries. Notably, from the perspective of intracluster compactness, scLLM-DSC facilitates a higher degree of aggregation within the manifold space, forming a more dense and robust cluster structure than the baselines. This dual enhancement in boundary separation and compactness validates the efficacy of cross-modal alignment in constraining the latent space to capture fine-grained cellular heterogeneity.

4.3 Biological Analysis

We conduct an in-depth analysis of the biological relevance of our clustering results following the standard evaluation pipeline of scCluBench [Xu et al., 2025c]. Taking the Mauro Pancreas dataset as a case study, we first identify the top 100 marker genes for each predicted cluster via SCANPY, with their expression specificity visualized in the tracksplot in Fig. 4(a). Subsequently, Fig. 4(b) illustrates the markeroverlap calculation process, which establishes a statistical foundation for biological identity assignments by calculating the intersection (overlap score) between predicted markers and gold-standard reference sets. Furthermore, in the Sankey diagram of Fig. 4(c), we compare two annotation paradigms: the purely numerical Best-Mapping (BM) strategy, which operates independently of external biological priors, and the biology-guided Marker-Overlap (MO) strategy. Analytical results demonstrate that while the BM strategy tends to "enforce mappings" for all nine potential cell types, the MO strategy exhibits superior biological rigor. Concretely, for cluster 1 in Fig. 4(b) mislabeled as " epsilon cell" by BM, the MO strategy deliberately labels the cluster as "pancreatic ductal cell" according to its highest overlap score, as shown in Fig. 4(c), ensuring higher fidelity cellular identification.

Table 4: Robustness to LLM Variants on Mauro Pancreas.

4.4 Ablation Study

Framework Component Study. To evaluate the contribution of each module, we compare scLLM-DSC with four variants on the Mauro Pancreas dataset by systematically removing Lalign (w/o align), LNCut (w/o NCut), LMSE (w/o MSE), and LKL (w/o KL). Results visualized in Fig. 5a reveal that scLLM-DSC consistently outperforms all ablation variants across all metrics, confirming that each component is vital for optimal performance. Notably, the significant performance degradation in w/o NCut and w/o align underscores the indispensable roles of both high-order structural manifolds and LLM-derived biological semantics. The performance gain demonstrates that scLLM-DSC effectively synergizes highorder topological structures with explicit biological semantics, leading to more robust and interpretable cell clustering.

Semantic Component Study. We evaluate the contribution of each semantic path by comparing scLLM-DSC with three variants: w/o AW (removing Abundance-Weighted Semantic Aggregation), w/o SC (removing Sequence-based Contextual Encoding), and w/o align (discarding both). Results in Fig. 5b show that while either path alone improves performance, their combination consistently yields the highest accuracy. This confirms that expression-weighted magnitudes and gene-order regulatory logic provide complementary biological insights into complex cellular states.

4.5 Robustness to LLMVariants

To assess the model's dependence on a specific language model, we investigate its robustness by substituting the default text-embedding-3-small with various LLM backbones of differing architectures and scales. As reported in Tab. 4, scLLM-DSC maintains remarkably stable performance across all evaluation metrics, with negligible fluctuations despite significant variations in the underlying embedding models. This stability indicates that scLLM-DSC is highly compatible with diverse LLM architectures, as the specific backbone choice is not the primary determinant of performance. Combined with Fig. 5, these results underscore that integrating LLMbased semantics remains indispensable for superior accuracy, demonstrating the broad applicability of our framework.

4.6 Parameter Sensitivity

We evaluate the impact of the balancing coefficient ω (Eq. 5) on clustering performance. As illustrated in Fig. 6, metrics including ACC, NMI, and ARI fluctuate with varying fusion ratios of abundance and context semantics. Optimal results across all metrics are achieved at ω = 0.5, confirming that the synergy of dual-path representations is essential for capturing comprehensive cell semantics. The performance decline at ω ∈ {0, 1} further proves that relying on a single semantic source is insufficient for maintaining peak clustering precision.

5 Conclusion

scLLM-DSC presents a novel paradigm that integrates LLMderived biological knowledge with deep structural clustering to address the persistent omission of semantic context in traditional scRNA-seq analysis. By synergizing functional gene priors with graph-guided topological encoding, our framework establishes a unified latent space that aligns numerical transcriptomic features with high-level biological meaning. Benchmarks confirm that this task-specific cross-modal alignment not only achieves state-of-the-art accuracy but also provides an interpretable foundation for mechanism-aware discovery. Looking ahead, we plan to scale this framework to atlas-level datasets and incorporate spatial transcriptomics to further decode complex tissue architectures.

Acknowledgements

This work is partially supported by the National Natural Science Foundation of China (Grant No. 92470204, 62406306, and 62406056) and the Guangdong Basic and Applied Basic Research Foundation (Grant No. 2024A1515140114).

Contribution Statement

Ping Xu and Pengjiang Li contributed equally to this work. Specifically, they led the project, provided theoretical support, and were responsible for overall model design, code implementation, experimental design, and paper writing. Tian Du, Zaitian Wang, Jiawei Gu, and Zhiyuan Ning provided guidance in solving complex problems and assisted with paper writing. Ziyue Qiao, Pengfei Wang, and Yuanchun Zhou provided valuable feedback on manuscript drafts. All authors reviewed and approved the final version.