Expanding the Category of Classifiers with LLM Supervision

1 Introduction

As the field evolves and data grows, new categories are often discovered or redefined, creating shifting demands for existing classifiers. Consequently, classifiers need to expand to accommodate emerging unseen categories, termed the classifier expansion task. Re-training with an expanded class set serves as a traditional solution, but requires significant image data collection and repetitive training, costly in many contexts.

The advent of zero-shot learning (ZSL) has inspired a zeroshot classifier expansion paradigm, which relies solely on images of seen classes [Xian et al., 2017; Wei et al., 2021; Xu et al., 2020]. Recent studies have further explored classifier expansion in completely image-free settings [Christensen et al., 2023; Yun et al., 2023]. These methods expand existing classifiers to recognize unseen classes by aligning expertconstructed attribute features with the classifier, thus integrating new classes into the visual embedding space without the need for any images. However, while these approaches reduce the image data requirements compared to re-training methods, they still depend on expert input, as attributes must be carefully designed and annotated, which keeps human costs involved [Xian et al., 2017]. We summarize the category expansion approaches, as illustrated in Figure 1.

∗Corresponding authors.

Inspired by the success of LLMs in reducing manual effort across various domains, we explore their potential to eliminate human dependency in classifier expansion. For this task, LLMs excel at generating detailed textual descriptions for a given class, offering valuable insights for "teaching" classifiers to recognize new labels. However, three key challenges remain. First, the descriptions generated by LLMs may not consistently maintain high quality for different classes, and often fail to offer the depth of information required to fully capture the characteristics of new classes. Second, the natural language output from LLMs frequently includes redundant or irrelevant details, requiring a refinement process to distill only the most essential features. Finally, the class features embedded in LLM-generated textual descriptions and those in the classifier are represented in different modalities, with significant differences in how these class representations are captured, making it complex to align them effectively.

To address these issues, this paper presents an LLM-driven classifier expansion framework that removes the need for manually designed attributes, termed Classifier Expansion with Multi-vIew LLM Knowledge (CEMIL). CEMIL consists of three sequential modules: LLM-based Descriptive Text Generation (DTG), Description Refinement with Contextual Filtering Attention (CFA), and Class Injection with Cross-modal Co-learning (CICC). The DTG module introduces an LLM reasoning flow to automatically derive comprehensive, hierarchical descriptions for classes. These multilevel descriptions are then embedded into refined representations by the CFA module, which employs high-level descriptions as contextual attention to filter out less relevant information. Finally, the CICC module aligns the refined embeddings with the classifier weights, enabling seamless cross-modal expansion of the classifier to new classes. Extensive experiments demonstrate that CEMIL, which operates solely with an LLM, consistently outperforms state-of-the-art ZSL methods that rely on expert-constructed attributes, across a variety of ZSL datasets. Figure 2 illustrates the framework of the CEMIL method. We summarize our contributions as follows:

• We propose a novel LLM-driven paradigm that enables the category expansion of existing classifiers without requiring any images or additional data. To the best of our knowledge, we are the first to use LLMs as a source of

supervision for human-free classifier expansion.

• We introduce CEMIL, a novel approach that: i) directs LLMs to deliver comprehensive and robust multi-view information, ii) extracts discriminative features from weak and noisy texts, and iii) effectively guides classifier expansion. This design stimulates and fully exploits the power of LLMs for cross-modal supervision.
• CEMIL achieves state-of-the-art performance on multiple image-free ZSL benchmarks, consistently demonstrating superior stability across various configurations. It holds the potential to reshape traditional ZSL tasks by introducing a new paradigm of LLM-based supervision.

2 RelatedWork

2.1 Category Expansion for Existing Classifiers

The task of classifier expansion involves extending an existing classifier to recognize new, unseen classes that were not defined during its training. As the application scenarios for classifiers evolve, this task has become increasingly critical and commonplace. However, the challenge lies in achieving effective expansion with minimal or cost-effective supervision, a key area of ongoing research primarily explored within the field of class-incremental learning (CIL) [Mittal et al., 2021; Wang et al., 2024]. Previous studies have extensively explored CIL in few-shot settings [Hersche et al., 2022; Zhou et al., 2024]. Unlike traditional re-training methods, few-shot CIL aims to expand classifiers using a smaller number of images from new classes, achieving notable success. As the available data continues to diminish in zero-shot settings, no image data from the new classes is available, and the ZSL models rely on additional side information to distinguish between classes. This information typically comes from expert-constructed class attributes [Li et al., 2023] or external textual sources, such as Wikipedia [Zhu et al., 2018].

The methods mentioned above typically rely on image data from seen classes, which is often unavailable in real-world scenarios. Consequently, some approaches explore classifier expansion in image-free settings by predicting and directly injecting the output layer parameters of new classes into the existing classifier, using expert-defined attributes to bridge the gap between seen and unseen classes. A straightforward approach involves using a multi-layer perceptron (MLP) to map visual classifier weights to attributes, which performs well in ZSL but tends to degrade in generalized settings. To address this limitation, Christensen et al. [Christensen et al., 2023] propose specific autoencoders for both attribute and weight spaces, regularizing the attribute-to-weight mapping and achieving strong performance on the image-free generalized ZSL task. Additionally, building on this image-free idea, other methods [Norouzi et al., 2014; Mensink et al., 2014; Aky¨urek et al., 2022; Xu et al., 2022] can also be adapted to expand classifiers in the image-free zero-shot setting.

2.2 LLM-driven Cross-modal Supervision

Zero-shot learning tasks require the use of auxiliary information to predict unseen categories. Most traditional approaches rely on expert-crafted attributes, such as manually designed numerical features [Lampert et al., 2013; Xian et al., 2017]. While these features are accurate and discriminative, they require manually defining attribute names and expert annotations for attribute values. This process is resourceintensive, particularly in fine-grained datasets [Wah et al., 2011]. Some methods use text descriptions from external sources like Wikipedia [Qiao et al., 2016; Zhu et al., 2018] or summary documents [Naeem et al., 2024] to reduce expert dependence, but this supervision is still manually constructed and limited in flexibility and information due to its disconnection from the ZSL task.

Recently, the rise of LLMs has shown great potential for enhancing few-shot or zero-shot learning tasks [Guo et al., 2023; Ban et al., 2025]. As a flexible knowledge source with advantages in both the quality and quantity of supervision, previous works have explored using LLMs for cost-effective knowledge expansion [Li et al., 2024; Wu et al., 2024]. An early work is I2MVFormer [Naeem et al., 2023], which uses multi-view prompting to encode LLM-supervised semantic embeddings for zero-shot image classification, while AdaptCLIPZS [Saha et al., 2024] extends this to pre-trained visionlanguage models (VLMs), achieving improved results. However, these methods still rely on image samples for training and fine-tuning. As an improvement, Liu et al. [Liu et al., 2024] explore training models solely on textual data by developing a cross-modal classifier with LLMs and mapping it to the visual modality, achieving zero-shot multi-label recognition with pre-trained VLMs. Despite these advances, these methods still heavily depend on pre-trained large-scale image encoders, such as CLIP, making them less suitable for scenarios involving existing image classifiers [Zhao et al., 2024].

In contrast, this paper uses only the cheapest LLM-based semantic knowledge to extend existing classifiers, without any image data or VLMs trained on large amounts of images.

This approach allows for classifier expansion with minimal annotation costs, data dependency, and domain bias.

3 Methodology

The entire CEMIL process consists of the following three stages: 1) Text Generation: Generate descriptive texts using a pre-trained LLM as the agent (DTG module). 2) Description Refinement: Distill and integrate global and multiview local descriptions using contextual filtering attention (CFA module). 3) Class Injection: Learn and inject classifier weights into the existing model through a cross-modal co-learning framework (CICC module). In this section, we first define the task and then introduce each stage in detail.

3.1 Task Formulation

We address the challenge of expanding a pre-trained classifier for unseen classes, without any image or attribute. Given a pre-trained classifier Φ: X → Cs, where X represents the image space and Cs denotes the set of seen classes, the objective of the Classifier Expansion is to expand the classifier's capability to classify a new class set C and get an expanded classifier ˆΦ: X → C. In the standard ZSL setting, the target set C is entirely disjoint from the seen classes, i.e. C ∩Cs = ∅. In the generalized ZSL setting, the target set C includes both the seen and unseen classes, i.e. C = Cs ∪ Cu. Previous ZSL works typically use an expert-based class attribute matrix A ∈ Rm×da , while this paper addresses the task using only an LLMM, and treat A as optional.

Without loss of generality, the classifier Φ can be divided into a feature extractor ω and a classification layer ψ, i.e., Φ = ψ(ω(·)). For a neural network classifier, the classification layer ψ is parameterized by a matrix, i.e., ψ ∈ Rdv×|Cs| , where dv is the embedding dimension of the classifier and the output dimension of ω. Since the task operates under the image-free setting, the goal is indeed to obtain an expanded classification layer parameter ˆ ψ ∈ Rdv×m.

3.2 LLM-based Descriptive Text Generation

We design a workflow for automatically generating descriptive texts from scratch, using an LLM as the agent. Without prior knowledge of the classifier, we first reason a comprehensive and effective set of views. Based on this, we employ a view-based batch querying strategy to generate descriptive and comparable multi-view texts for each class.

Structured Hierarchical View Reasoning

View generation can be complex for LLMs, as class identification often requires fine-grained views, which can lead to omissions or overlap, especially when the view number is large. Considering the limited capacity of LLMs, we aim to decompose the view generation task into smaller subtasks, enabling the LLM to tackle them in a systematic manner. Specifically, we guide the LLM to first generate coarsegrained perspectives (e.g., physical traits, behavioral traits) and then refine them into fine-grained details (e.g., fur color, feeding habits). This hierarchical divide-and-conquer approach reduces the LLM's capacity demand in a single pass.

To further enhance the quality, we prompt the LLM to act as a domain expert, leveraging its specialized knowledge, and provide structured examples to harness its in-context learning ability for better task understanding. These techniques are also employed in other LLM reasoning scenarios. Integrating these components, we design a meta prompt Pmeta to query the LLM for n0 rich views, forming an initial set of candidate

where T denotes the set of textual segments.

To distill an effective view set, we select views through an iterative self-verification process. In this process, the LLM reassesses the initial view set V (0) from an evaluator's perspective, verifying its relevance and filtering views that are genuinely beneficial for visual class identification. The verification process iteratively refines the view set V (t) using the verification prompt Pverify. At each iteration t, an updated set V (t+1) is produced, and the process stops when the set size stabilizes, indicating no further reduction:

where s = 0, 1, . . . , S-1, and S denotes the final iteration s. The converged view set is denoted as V = V (S) , containing n views. This iterative self-verification ensures a robust view set, forming a reliable foundation for downstream tasks.

View-based Batch Querying Strategy

Based on the class set C and the view set V, we can construct the class-view description matrix S ∈ Tm×n , where m is the size of C, and each element in S corresponds to a textual description. In terms of querying the LLM, a simple point-to-point query involves querying each class-view pair individually to populate the matrix S, and class-based querying focuses on querying one class across all views in a single round. However, due to the inherent randomness and quasiindependence of LLM outputs, both strategies can introduce biases across different classes under the same view, undermining inter-class comparability. Since our downstream tasks prioritize class-level comparability, we adopt the View-based Batch Querying Strategy. This approach retrieves descriptions for all classes under a single view in each round, preserving inter-class comparability across views.

We design a prompt Pmain specifically tailored for viewbased querying. The process can be expressed as follows:

Finally, S is encoded by a pre-trained embedding model:

where dt is the embedding dimension of the encoder. Opensource LLMs can also serve as embedding models, and we will validate their effectiveness in the experiments.

Inspired by [Liu et al., 2024], we construct a global summary for each class using a global prompt Pglobal to extract comprehensive feature information. These features are embedded similarly to the multi-view descriptions, expressed as:

where G represents the global representation of the classes. All parameters in both the LLM and embedding model remain frozen throughout the training process.

3.3 Description Refinement with Contextual Filtering Attention

We seek to extract task-relevant, effective class representations from the diverse and noisy descriptions derived from global and multiple local views. This is achieved through a contextual filtering attention module.

For each class c in C, we begin by projecting its global description Gc and multi-view local description Hc into the standard representations of the attention mechanism using linear transformations. Since Gc contains only global view, we expand its dimension into Rn×dt by copying the second dimension for n times in advance. Let WQ,WK,WV be three MLPs that map the inputs to the query, key, and value representations, respectively. Specifically, the query, key, and value are defined as Q = WQ(Gc), K = WK(Hc), and V = WV (Hc), respectively. Here, Q, K, V ∈ Rn×de , where de is the embedding dimension. In this design, the global description, containing high-level class information (e.g., key features), serves as the query, guiding the alignment of multiview local descriptions.

The energy score e ∈ Rn×n is computed based on a combination function f(·), which incorporates both the cosine similarity and the Euclidean distance between the query and key representations, formulated as follows:

where CosSim(·) denotes cosine similarity and ||Q - K||2 represents the Euclidean distance between the Q and K. The cosine similarity and Euclidean distance complement each other, and their product simultaneously captures the global direction of the vectors and their local spatial differences, enabling better handling of complex relationships. The energy score reflects both the similarity and the spatial discrepancy between the query and key representations, leading to improved alignment of multi-view descriptions.

To transform the energy score e into a probability distribution, it is normalized into attention weights γ using a distribution function. In this case, we use the Softmax function to ensure that the weights sum to 1 and reflect the relative importance of each element in the sequence.

Finally, we aggregate the value matrix V by performing matrix multiplication with the attention weights γ to obtain the weighted features:

The weighted feature T is a merged text-based embedding that captures the most discriminative features from the multiview descriptions. Additionally, if an optional attribute A is available, it can be incorporated into the embedding representation T by concatenating it along the last dimension after dense embedding, thereby enhancing overall performance.

3.4 Class Injection with Cross-modal Co-learning

The fused text-based LLM-generated embedding, together with the vision-based classifier weight vector, is fed into a dual-autoencoder co-learning network. The network maps the text-based and vision-based information into a shared latent space Z, enabling mutual supervision and integration of information from both sources. Specifically, the encoder for the vision-based embeddings is defined as Ev : ψ → Z, with its corresponding decoder Dv : Z → ψ. Similarly, the encoder for the text-based embeddings is Et : T →Z, with the decoder Dt : Z → T. To enable the network to focus on the angular alignment of the injected weights, we adopt cosine distance as the distance metric d(·).

During the training stage, for each class, we strive to map both modalities of data into the same latent space by optimizing two types of loss: reconstruction loss within each embedding and cross-modal loss between both embeddings.

The reconstruction loss of the vision-based embedding aims to ensure the stability of the classifier weight embedding of seen classes, formulated as:

Similarly, the reconstruction loss for the text-based embedding, applied to both seen and unseen classes, is defined as:

where an additional term ∑c∈C d(DA t (Ev(T c )),A) can be added to LT→T if the class attribute is available.

The design of reconstruction losses ensures proper encoding in both text-based and vision-based latent spaces. However, enabling interactive learning between these two modalities requires alignment of their latent spaces. We achieve this goal through two cross-modal loss functions:

where an additional term ∑c∈Cs d(DA t (Ev(ψc )),Ac ) can be added to LsV→T if the class attribute is available. The design of cross-modal loss facilitates latent space alignment in a bidirectional learning manner.

Finally, we add all reconstruction and cross-modal losses to obtain the total loss function L:

During the inference stage, we use the text-based embeddings to infer the expanded classifier weights ˆ ψ, as follows:

where ˆ Dv is the trained vision-based decoder, and ˆ Et is the trained text-based encoder. The classifier expansion is completed once its weights are updated.

4 Experimental Study

Experiments on three common datasets are conducted. The results demonstrate the effectiveness and strong stability of CEMIL in both complementary and substitutive scenarios.

4.1 Benchmark Protocol

Datasets. Methods are evaluated on three widely used datasets: 1) AWA2 [Xian et al., 2018], an animal classification dataset featuring 50 mammal species; 2) CUB [Wah et al., 2011], a dataset containing 200 bird species; and 3) SUN [Patterson et al., 2014], a scene recognition dataset with 717 categories. Each dataset is accompanied by expert-constructed class attributes and is divided into seen and unseen classes based on the splitting scheme in [Xian et al., 2017]. Following the image-free setting, we use only the part of class attributes here, without any images from these datasets.

Implementation Details. For each dataset, we pre-train a ResNet-101 [He et al., 2016] using images and labels from only the seen classes, following [Xian et al., 2019], as the base classifier for expansion. We utilize GPT-4o [OpenAI, 2023] as the LLM and the text encoder of CLIP [Radford et al., 2021] as the embedding model, while subsequent analysis will demonstrate that this configuration has minimal impact on overall performance. The initial expected view number is set to 50. Each encoder is a single-layer MLP, while each decoder is a two-layer MLP with a hidden dimension 4096. The dimensions of the attention vectors are 2048. Neural network parameters are initialized randomly from a standard normal distribution. The Adam optimizer is used for training, with up to 500 epochs and an early stopping strategy. The learning rate is set to 1e-5, with batch sizes configured as 10, 16, and 32 for AWA2, CUB, and SUN, respectively. Experiments are conducted on an Nvidia GeForce RTX 4090 24GB GPU.

Evaluation Metrics. We perform evaluations of the methods under both standard and generalized ZSL settings. For standard ZSL, Top-1 accuracy is used as the metric, denoted as T. In generalized ZSL (GZSL), we calculate the accuracy for both unseen and seen classes, denoted as u and s, and compute their harmonic mean H by 2 × (s × u)/(s + u).

4.2 Comparison with State-of-the-Arts

The proposed CEMIL is compared with six image-free ZSL baselines: 1) MLP utilizes a two-layer neural network to map the class attributes to their corresponding weight vectors. 2) ConSE [Norouzi et al., 2014] generates representation for unseen samples by computing a weighted sum of seen class embeddings, with the unseen sample's predicted probabilities as weights. 3) COSTA [Mensink et al., 2014] also employs a weighted sum approach but uses co-occurrence statistics between classes as weights. 4) SubReg [Aky¨urek et al., 2022] applies subspace regularization to keep unseen class weight vectors close to the subspace of seen classes, reducing catastrophic forgetting in CIL. 5) VGSE [Xu et al., 2022] clusters local regions of seen classes by visual similarity and links these clusters to unseen classes via optimizing a class attributes similarity matrix. 6) ICIS [Christensen et al., 2023] uses two separate encoder-decoders to predict unseen weights: one for class attributes and one for weight vectors.

The performance of these methods is evaluated across three scenarios: 1) Expert-constructed Attribute only: Utilizes attributes defined and annotated by domain experts, as provided in the ZSL datasets. 2) LLM-based Supervision only: Replaces expert-constructed attributes with multi-view descriptions generated using the proposed LLM-driven Robust Feature Enhancement method. For baseline methods, we ensure fairness by using the same multi-view embeddings as in CEMIL and applying mean pooling along the view to match the input shape of the attributes. 3) Expert-constructed Attribute + LLM: Combines both expert-constructed attributes and LLM-based supervision through concatenation.

Table 1 provides a comprehensive comparison of standard and generalized ZSL metrics across various methods. As demonstrated, CEMIL consistently outperforms state-of-theart baselines across both scenarios and all three benchmarks, achieving significant improvements (mostly over 10%) in both standard and generalized ZSL settings. The performance of CEMIL, using both attribute and LLM supervision, reaches optimal levels, showcasing its effectiveness to complement existing expert-constructed attribute-based methods.

Notably, the proposed CEMIL, when using a more flexible LLM-based supervision, can achieve and even surpass SOTA methods that rely on expert-constructed attributes (i.e., for ZSL: +6.9% on AWA2, +1.9% on CUB, +6.4% on SUN; for GZSL: +11.0% on AWA2, +4.6% on SUN). This highlights its ability to achieve superior performance with costeffective LLMs, replacing traditional attribute-based methods, and underscores its potential to shift ZSL tasks from expert-dependent to LLM-driven approaches.

4.3 Ablation Study

To evaluate the contribution of key components in the proposed CEMIL framework, we perform a series of ablation experiments. The experiments are conducted by progressively modifying the full framework through the following steps: 1) Replacing hierarchical view reasoning in the DTG module with a plain prompt, and removing the process of iterative view selection. 2) Substituting the view-based text querying in the DTG module with a class-based querying approach. 3) Replacing the CFA module with a simplified version that integrates multi-view descriptions via mean pooling, concatenated with class attributes. 4) Replacing the CICC module with a single encoder-decoder architecture. 5) Removing the LLM supervision and relying solely on class attributes. As shown in Table 2, removing any module results in a performance decline, highlighting the importance and effectiveness of each proposed component throughout the workflow.

4.4 Empirical Analysis

Effect of the Number of Views and Seen Classes

In this experiment, we investigate the impact of the number of views and seen classes on model performance, as illustrated in Figure 3. The maximum number of views generated by the LLM is set to 50, with the number of views gradually increasing from 0. Additionally, we experiment with varying proportions of seen classes used for training, ranging from 20% to 100%. As the number of views increases, all performance metrics improve, and the rate of improvement slows beyond 20 views. Similarly, a larger number of seen classes has a positive impact on the final performance.

(a) ZSL Top-1 Accuracy (T)

(b) GZSL Harmonic Mean (H)

Figure 4: This figure illustrates the improvement in H achieved by combining LLMs with the proposed framework, highlighting the enhancements of CEMIL across various base knowledge sources.

Ability to Enhance Existing Knowledge Sources

To demonstrate the effectiveness of CEMIL in improving the efficiency of expert- or semi-expert-based sources, we conduct comparative experiments on different sources and datasets. We evaluate three knowledge sources: expertconstructed class attributes (Att), ConceptNet features [Speer and Lowry-Duda, 2017] (CN), and Wikipedia features [Yamada et al., 2020] (W2V). Each experiment is conducted both with and without the assistance of the LLM, and the results are presented in Figure 4. Across all datasets and knowledge sources, CEMIL significantly improves the harmonic mean of zero-shot classifier expansion, highlighting its stable ability to enhance existing supervision sources.

Applicability of Different LLMs and Embedding Models To validate the applicability of CEMIL across different LLMs, we replaced both the LLM and embedding components with different models. For the LLM, we evaluated four models, encompassing both open-source and proprietary options: GPT-4o [OpenAI, 2023], GPT-4o mini, LLaMA3.1 [Touvron et al., 2023], and Qwen-plus [Team, 2023]. For the embedding component, we explore the text encoder of CLIP [Radford et al., 2021], SBERT [Reimers and Gurevych, 2019], and two open-source LLM-based embedding methods: LLaMA-3.1-8b and Qwen-2.5-7b. The LLM-based embedding is implemented by applying mean pooling to the vector from the last layer of the LLM, given the input text.

As shown in Table 3, all tested configurations of CEMIL perform well and consistently outperform previous attributebased approaches. This demonstrates the robustness and broad applicability of CEMIL across various embedding models and LLM architectures. The configuration using LLaMA for both the LLM and the embedding model represents a scenario where classifier expansion is achieved solely through an offline open-source LLM, offering a highly flexible and practical choice for real-world applications.

5 Conclusion

This paper proposes CEMIL, a novel framework for expanding existing classifiers without the need for any images or human effort. CEMIL uses a structural two-stage strategy to deliver robust LLM supervision, integrates multi-view descriptions with contextual filtering attention, and employs a cross-modal co-learning framework to expand the classifier. Experiments show that CEMIL achieves state-of-the-art performance across multiple ZSL benchmarks in both standard and generalized image-free ZSL settings. The entire framework can be completed with only a pre-trained LLM, which can reshape the existing attribute-based ZSL paradigm in both complementary and substitutive scenarios. This work paves the way for flexible, cost-effective model adaptation to newly emerging classes in a fully automated manner.

Acknowledgments

This research was supported in part by the National Key R&D Program of China (No. 2021ZD0111700), in part by the National Nature Science Foundation of China (No. 62406302, 62137002, 62176245, 62206261), in part by the Natural Science Foundation of Anhui province (No. 2408085QF195), in part by the Fundamental Research Funds for the Central Universities under Grant (No. WK2150110035), in part by Special Foundation for Science and Technology Innovation and Entrepreneurship of CCTEG (No. 2020-2-TD-CXY006).