Reducing Bias and Variance: Generative Semantic Guidance and Bi-Layer Ensemble for Image Clustering

1 Introduction

The goal of image clustering is to partition a dataset into multiple disjoint subsets by learning from unlabeled image data. A core aspect of this task is constructing and leveraging prior knowledge to form effective supervisory signals.

Based on the nature of the prior information, image clustering methods can be broadly categorized into several types. Classical clustering approaches rely heavily on assumptions about data distributions, including prototype-based methods [McQueen, 1967], density-based methods [Ester et al., 1996], and hierarchical methods [Dasgupta, 2002]. Deep clustering methods leverage powerful representation learning to extract highly discriminative features as prior information [Xie et al., 2016; Caron et al., 2018]. Advances in self-supervised learning have further shifted clustering paradigms toward supervision signals derived from data augmentation [Li et al., 2021b]. Nevertheless, these approaches rely exclusively on intrinsic information within the data to guide clustering. As a result, the supervisory signals are inherently constrained by the data itself [Liu et al., 2024a], making further performance improvements increasingly difficult in practice.

Inspired by the zero-shot classification paradigm introduced by CLIP [Radford et al., 2021], recent studies have shifted the source of supervision from internal priors to external semantic descriptions. Methods such as SIC [Cai et al., 2023] and TAC [Li et al., 2023b] typically adopt predefined vocabularies (e.g., WordNet) and employ pre-trained vision-language models to embed images and text into a shared representation space. Then, images are matched with the most relevant textual labels to generate semantic descriptions, which serve as external semantic priors to guide the clustering process. Most of these methods construct semantic prior information by matching images with textual libraries. However, due to the limited coverage of predefined vocabularies, forced alignment between images and text may lead to semantic inconsistency, particularly when complex visual semantics surpass the representational capacity of the fixed lexical set. Therefore, these approaches may be constrained by their reliance on matching-based semantic prior construction.

Moreover, the strategy of introducing semantic priors is essentially to improve the clustering performance by reducing the bias. In machine learning theory, as is well known, estimation error arises from the joint effects of bias, variance, and noise. Therefore, explicitly accounting for variance reduction during clustering is expected to further improve performance.

Based on the discussions above, we propose Image Clustering based on Generative Semantic Guidance and Bi-Layer Ensemble, noted as GSEC. This framework jointly improves clustering performance by addressing both bias and variance. To reduce bias, we introduce a generative semantic prior construction strategy that overcomes the limitations of matching-based semantic priors. This strategy leverages Multimodal Large Language Models to generate semantic descriptions and synthesizes corresponding semantic representations for each image via weighted averaging. To reduce variance, we incorporate ensemble learning through a bi-layer ensemble strategy, comprising an inner-layer ensemble and an outer-layer ensemble. The inner layer extracts domain information from both semantic and image modalities and performs internal ensemble via BatchEnsemble to achieve semantically guided image clustering. The outer layer employs an alignment-based ensemble mechanism to align the inner outputs with the outer outputs, with the aligned result serving as the final clustering outcome.

Our contributions are summarized as follows:

• Generative Semantic Guidance: We propose a bias reduction strategy that leverages multimodal large language model to generate adaptive semantic descriptions for images, thereby reducing the inconsistency of the match-based method.
• Bi-Layer Ensemble Mechanism: We design a bi-layer ensemble framework to mitigate variance. The inner ensemble integrates multimodal information via BatchEnsemble, and the outer ensemble aligns inner predictions with external outputs, thereby enhancing robustness and stability.
• Comprehensive Empirical Validation: We conduct experiments on benchmark datasets, demonstrating that GSEC significantly outperforms the reference methods. The bias-variance decomposition analysis confirms that our approach effectively reduces both bias and variance.

2 RelatedWorks

In this section, we briefly review image clustering and ensemble learning.

2.1 Image Clustering

The evolution of image clustering methodologies has progressively incorporated richer forms of prior knowledge to construct more effective supervisory signals [Ren et al., 2024]. In the following, we review these methods according to their primary sources of supervision: internal prior information and external semantic information.

The development of the internal prior information-based clustering algorithms can be broadly divided into three stages. Early classical clustering methods typically rely on low-level visual semantics: K-means [McQueen, 1967] groups data by minimizing distances to cluster centers, while DBSCAN [Ester et al., 1996] identifies arbitrarily shaped clusters based on local density. With the advent of deep learning, deep clustering methods emerged: DEC [Xie et al., 2016] learns feature representations via autoencoders and jointly optimizes a clustering objective; ClusterGAN [Mukherjee et al., 2019] implements clustering through adversarial learning. Subsequently, self-supervised approaches were developed: DeepCluster [Caron et al., 2018] iteratively uses clustering outcomes as pseudo-labels for training, and CC [Li et al., 2021b]

further integrates contrastive learning with clustering objectives to enhance performance.

Vision-language pre-training models such as CLIP [Radford et al., 2021] have shifted image clustering from relying on internal priors to exploiting external semantic information. Based on CLIP, SIC [Cai et al., 2023] and TAC [Li et al., 2023b] enhance clustering by modeling image-text geometric consistency and incorporating WordNet semantics with cross-modal distillation, respectively. GradNorm [Peng et al., 2025] further improves text-assisted image clustering by selecting semantically relevant nouns from unlabeled data. [Wang et al., 2026] extend text-guided learning to visual adaptation through deep manifold constraints and neighborhood propagation.

Despite these advancements, existing multimodal clustering methods remain fundamentally limited by their reliance on matching mechanisms within a constrained vocabulary space. Due to the restricted expressiveness of predefined vocabularies, forced alignment between images and textual labels inevitably leads to semantic inconsistency when complex visual semantics exceed the coverage of the fixed lexical set.

Moreover, existing strategies that incorporate semantic priors primarily target bias reduction to improve clustering. It is important to note that estimation error stems from the combined effects of bias, variance, and noise. Therefore, to achieve more robust and stable clustering, addressing variance reduction is also essential. Ensemble learning serves as a key technique for variance reduction in machine learning.

2.2 Ensemble Learning

Ensemble learning addresses complex tasks by constructing and combining multiple learners, and has been widely studied as an effective strategy for improving robustness and stability through the integration of complementary results.

In clustering, different initializations, data views, or similarity measures may produce inconsistent partitions, making the final results sensitive to randomness and data perturbations. To alleviate this issue, representative studies have explored multigranulation information fusion [Li et al., 2017], sample-stability-based clustering ensemble [Li et al., 2019], cluster quality evaluation and selective clustering ensemble [Li et al., 2018], and space-structure-based clustering for mixed-type data [Li et al., 2022]. Reliable evaluation has also been investigated by mitigating random consistency in accuracy measures [Wang et al., 2020] and analyzing the generalization performance of pure accuracy for selective ensemble learning [Wang et al., 2022]. Recent methods further enhance clustering ensemble by designing structured fusion mechanisms, including the growing tree model GOT [Li et al., 2021a], fuzzy ensemble clustering with selfcoassociation and prototype propagation [Li et al., 2023a], and k-HyperEdge medoids [Li et al., 2025a].

These studies show that ensemble learning can exploit complementary information from multiple learners, partitions, samples, clusters, or semantic structures to produce more stable and reliable results. By aggregating diverse yet correlated outputs, ensemble methods can effectively suppress the fluctuation of individual predictions and thereby reduce the variance of the final decision.

3 Method

In this section, we propose an image clustering method based on generative semantic guidance and bi-Layer ensemble (GSEC), with its overall framework illustrated in Figure 1. Firstly, we introduce the generation of semantic embeddings using Multimodal Large Language Models. Secondly, we elaborate on the operational mechanism of the designed bilayer ensemble framework.

3.1 Generative Semantic Embedding

To address the limitation of constrained semantic matching space, we leverage Multimodal Large Language Models (MLLMs) to generate semantically enriched descriptions through reasoning. The procedure is detailed as follows.

Let V = {Ii}ni=1 denote the image set, where n is the number of samples. Using an image encoder Ev, we first extract image embeddings vi = Ev(Ii) ∈ Rd , with d representing the embedding dimensionality.

To uncover latent semantic structures, we perform Kmeans clustering on all image embeddings {vi}ni=1 . Following the practice in [Peng et al., 2025; Li et al., 2023b], the number of clusters is set as C = max { n 300 , 3K}, where K denotes the expected number of clusters. Images within each cluster are then sorted by the distance between their embeddings and the corresponding cluster center. From each cluster, we uniformly select up to five representative images according to this sorted order. If a cluster contains fewer than five images, all of them are selected.

These selected images are fed into the Multimodal Large Language Model using a natural-language prompt designed to capture core visual semantics. The prompt is:" Identify and describe the main object in this image. Respond with the format:' This image contains a [object] characterized by [attribute1], [attribute2], and [attribute3]'".

The generated class-level descriptions Tk are encoded by the CLIP text encoder Et into class embeddings ¯ tj = Et(Tk). The similarity between each image embedding vi and every class text embedding {¯ tj}Mj=1 is then computed to derive an image-specific text embedding ti, where M is the number of selected samples.

Specifically, we first compute the conditional probability of the j-th class text embedding given image vi:

where sim(·, ·) denotes cosine similarity, and ˜ τ is a temperature parameter that controls the smoothness of the distribution. Subsequently, the final text counterpart ti is obtained as a weighted average of all classes embedding:

Here, ti represents the text embedding of the i-th sample.

3.2 Bi-Layer Ensemble

To mitigate variance, we design a bi-layer ensemble framework: the inner layer employs a BatchEnsemble integrator for cross-modal clustering, and the outer layer further aligns and integrates the inner outputs with external predictions through an alignment-based ensemble mechanism.

Inner Ensemble

At the inner layer, we construct an integrator based on the BatchEnsemble method to efficiently integrate cross-modal data. This integrator consists of two independent branches for image and text modalities, each comprising a set of shared weights and member-specific low-rank modulation parameters.

The output of the k-th base model in the BatchEnsemble integrator is given by:

where W denotes the shared weight matrix across all members, rk and sk are learnable rank-1 vectors specific to the k-th member, ⊙ represents the element-wise multiplication, and bk is the bias term for the k-th member.

We denote the BatchEnsemble integrators for the image and text modalities as lvk (x) and ltk (x), respectively. Let vN i be a random neighbor of vi, and tNi be a random neighbor of ti. We process vi and vN i through the image BatchEnsemble integrator, and ti and tNi through the text BatchEnsemble integrator. As for yv i and yt i :

where m is the ensemble size, yv i and yt i represent the soft assignments for the i-th image and text, respectively. As for yv,N i and yt,N i :

where yv,N i and yt,N i denote the neighbor soft assignments for the i-th image and text, respectively.

To achieve mutual guidance between the text and image modalities during clustering, we introduce the Cross-Modal Mutual Distillation loss:

where DKL denotes the KL divergence. This loss serves a dual purpose: it enhances the discriminability of cluster boundaries by reducing inter-cluster similarity, while also enabling bidirectional alignment between each image and the neighbors of its corresponding text, and between each text and the neighbors of its corresponding image.

To stabilize training and encourage the model to produce more confident cluster assignments, we introduce the confidence loss:

To prevent the collapse of all samples into only a few clusters, we adopt the balance loss:

where K is the number of clusters. ¯ yv and ¯ yt denote the cluster assignment distributions in the image and text modalities, respectively, which are computed as:

Then, the inner ensemble aims to optimize the model parameters by minimizing the following comprehensive loss function:

Outer Ensemble

After the inner-layer ensemble training converges, we compute the average of the outputs from the image and text modalities:

where ˆ yi denotes the output of the inner ensemble.

In the outer ensemble stage, we train a task encoder ϕ that takes the concatenated image and text embeddings [vi; ti] as input, with its output defined as:

To align the predictions of the inner ensemble with the outputs of the outer task encoder, we adopt the cross-entropy loss as the alignment objective:

The outer ensemble optimizes the parameters ϕ by minimizing the following comprehensive loss:

where ¯ p = 1n ∑ni=1 ϕ([vi; ti]) is the average prediction distribution, and H(¯ p) denotes its entropy.

After the outer ensemble training converges, the final cluster assignment for each sample is obtained from the output of the task encoder. This decision mechanism integrates the multimodal alignment information from the inner ensemble outputs with the comprehensive inference of the task encoder, which is expected to generate a stable and consistent clustering result.

4 Experiments

In this section, we validate GSEC's performance through benchmark comparisons. Additionally, we conduct biasvariance decomposition analysis to verify the effectiveness of our method, along with ablation studies, hyperparameter analysis, convergence analysis, practicality analysis, and visualization analysis (In Appendix [Li et al., 2026]).

4.1 Experiment Setup

Datasets To evaluate the effectiveness of our proposed method, we conduct experiments on 11 widely used benchmark datasets. The detailed information of these datasets is summarized in Table 1.

Evaluation To evaluate the clustering performance, we employ three widely adopted metrics: Accuracy (ACC), Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI). For all three metrics, higher values indicate better performance.

Implementation Details We utilize the pre-trained CLIPViT-B/32 model to extract image and text embeddings. For semantic descriptions, we employ Llama-3.2-Vision-11B to generate descriptions from representative images, which are then standardized into a unified format. The learning rates for both the internal integrator and the task encoder are set to 0.001, and the number of ensemble is set to 24. The batch size is fixed at 1024, with the exception of ImageNet-1K, which uses a batch size of 8192. All experiments were conducted on a single NVIDIA A6000 GPU.

4.2 Comparison with Baseline Methods

To comprehensively assess the effectiveness of GSEC, we benchmark it against a wide range of state-of-the-art algorithms, categorized into two groups: (1) Unimodal Clustering Methods, including DEC [Xie et al., 2016], DAC [Chang et al., 2017], JULE [Yang et al., 2016], DCCM [Wu et al., 2019], IIC [Ji et al., 2019], SCAN [Van Gansbeke et al., 2020], TCL [Tao et al., 2021], TCC [Shen et al., 2021], GCC [Zhong et al., 2021], CRLC [Do et al., 2021], RPSC [Liu et al., 2024b], PICA [Huang et al., 2020], SPICE [Niu et al., 2022], DivClust [Metaxas et al., 2023], and LFSS [Li et al., 2025b]; (2) Multimodal Clustering Methods, such as TAC [Li et al., 2023b], CLIP (K-means), SIC [Cai et al., 2023], GradNorm [Peng et al., 2025], and CAE [Zhu et al., 2026].

We evaluate our proposed GSEC on the first five datasets listed in Table 1 and compare it with the above mentioned 18 image clustering baselines. As shown in Table 2, our method consistently outperforms these competitors. Furthermore, we test GSEC on the challenging large-scale ImageNet-1K dataset and compare it with four leading multimodal methods. As presented in Table 5, our approach retains its superiority and delivers the best performance.

4.3 Reduction of Bias-Variance Analysis

To evaluate the effectiveness of the proposed method in reducing bias and variance, we first generate 10 base datasets via resampling with the same sample size for each dataset. Each of these datasets is used to train a model. Bias is defined as the average deviation between the model's predictions and the ground truth, while variance quantifies the dispersion across predictions from individual models.

We test the bias-variance performance of the following five configurations. Image: using only image features with the bi-layer linear architecture; Image+Ensemble: using only image features with the bi-layer ensemble framework; Image+M-Text: feeding image features and matchingbased textual features into a bi-layer linear architecture; Image+G-Text: feeding image features and generative textual features into the bi-layer linear architecture; GSEC: feeding image features and generative textual features into the bi-layer ensemble framework.

The experimental results are illustrated in Figure 2 and Figure 6 (in Appendix [Li et al., 2026]). The following observations can be observed: (1) Comparing Image and Image+Ensemble, the ensemble strategy effectively reduces variance. (2) Comparing Image+M-Text and Image+G-Text, generative textual guidance helps reduce bias but introduces additional variance. (3) Compared with other methods, GSEC simultaneously reduces bias and variance through generative textual guidance and ensemble strategy, yielding superior and more stable clustering performance.

4.4 Ablation Analysis

To verify the effectiveness of each component in the proposed method, we conduct ablation studies to examine the impact of generative semantic embeddings and the bi-layer ensemble design on the performance of GSEC. Experiments are carried out across multiple pre-trained backbones, including CLIP-ViT-B/32, CLIP-RN50x4, CLIP-RN101, and CLIP-RN50. Performance is evaluated using Accuracy.

Effectiveness of the Generative Semantic We derive semantic prior information from two distinct sources: the Llama-3.2-Vision-11B MLLM for generative semantics (denoted as G-Text) and the WordNet for matching semantics (denoted as M-Text). These textual descriptions are encoded by separate feature extractors and subsequently processed through the same bi-layer ensemble framework for clustering. As shown in Table 3, G-text consistently outperforms M-text across all tested backbones.

Effectiveness of the Bi-layer Ensemble We compare the proposed bi-layer ensemble framework with a bi-layer linear architecture under identical image feature inputs. As reported in Table 4, our ensemble-based approach yields consistent performance gains on ten benchmark datasets across the four different backbones.

4.5 Hyperparameter Analysis

Analysis of the number of ensembles. We analyze the impact of the ensemble size on clustering performance and computational efficiency. As illustrated in Figure 3, the runtime increases with the number of ensemble members, and an ensemble size of 24 achieves a favorable balance between performance gains and computational cost.

Analysis of Internal and External Learning Rates. We investigate the effect of different combinations of inner and outer learning rates on clustering performance, aiming to identify an optimal configuration that ensures stable clustering accuracy. As shown in Figure 4, the model attains the best overall performance across all ten benchmark datasets when both the inner and outer learning rates are set to 0.001.

4.6 Convergence Analysis

To examine the training convergence of GSEC, we visualize the evolution of both inner and outer losses over training epochs. As shown in Figure 5, both losses exhibit a clear downward trend and eventually converge, demonstrating stable optimization.

4.7 Practicality Analysis

To assess the efficiency of semantic querying, we report the total query tokens and average latency on CIFAR10 and STL10. As shown in Table 6, our method incurs limited query overhead and achieves practical response efficiency, with latency of only a few seconds.

5 Conclusion

This paper proposes GSEC (Image Clustering based on Generative Semantic Guidance and Bi-Layer Ensemble), a novel framework that jointly reduces bias and variance in image clustering. To overcome the limitations of vocabulary-matching-based semantics, GSEC leverages Multimodal Large Language Models to generate adaptive semantic descriptions, which mitigate semantic inconsistency and reduce bias. Simultaneously, a bi-layer ensemble mechanism is introduced to lower variance and improve robustness. This ensemble consists of an inner layer that integrates cross-modal information via BatchEnsemble and an outer layer that aligns inner and outer predictions through an alignment-based ensemble. Experiments on 6 benchmark datasets demonstrate the effectiveness of GSEC in image clustering. Further bias-variance decomposition confirms that GSEC reduces both bias and variance concurrently, highlighting the importance of jointly optimizing these two factors for improved clustering performance.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Nos. 62441239, 62476160, T2495251, 62441239, 62306170, U24A20253, 62136005), the Special Fund for Science and Technology Innovation Teams of Shanxi Province (No. 202304051001001).