From Sparse to Complete: Semantic Understanding Based on Stroke Evolution in On-the-fly Sketch-based Image Retrieval

1 Introduction

With the widespread adoption of interactive touchscreen devices, sketch-based image retrieval (SBIR) has emerged as an accessible and practical modality. In particular, the advent of on-the-fly frameworks[Bhunia et al., 2020; Liang et al., 2021; Liu et al., 2022; Dai et al., 2024b] has significantly lowered the entry barriers for users, enabling real-time sketching and retrieval, thereby reducing the required interaction time and obviates the need for complete sketches. However, a fundamental challenge in the early stages of on-the-fly SBIR is differentiating between keystrokes indicative of user intent and those that are unintentional or noisy strokes. To convey specific shapes, structures, or details, users may draw strokes that deviate from typical features expected by the model. Although these strokes may carry significant semantic information for the user, existing models may typically misclassify them as noise, negatively affecting the understanding of sketch intent and recognition accuracy. For example, as shown in Fig. 1, when users aim to represent the textural details of shoelaces, the drawn strokes may exhibit slight curvature or incomplete closures. These strokes encode significant semantic information about the texture. However, deep learning models that primarily rely on statistical regularities for feature matching may consider these strokes as noise that deviates from established category features. This phenomenon is particularly pronounced in objects with intricate details.

Recent studies have attempted to address noise strokes in SBIR [Koley et al., 2024b; Bandyopadhyay et al., 2024]. For example, Sain et al. [Sain et al., 2021] proposed a styleagnostic model, Bhunia et al. [Bhunia et al., 2022a] developed a stroke subset selector to filter noisy strokes, and Koley et al. [Koley et al., 2024a] focused on handling abstract strokes. However, these approaches are largely oriented toward complete sketches. In the early stages of onthe-fly retrieval, where models receive only sparse strokes and lack contextual information, the identification of noise strokes and the extraction of effective features become significantly more challenging. The core challenges in on-thefly SBIR are twofold. (1) Accurately determining whether a given stroke is integral to the target features or an irrelevant distractor is challenging, particularly before the key features of the target object are fully rendered. This challenge is particularly pronounced for detail-rich objects such as facial sketches. (2) Extracting semantic features from sparse, incomplete strokes in the initial stages of sketching presents another challenge, as it is essential for effective matching during early retrieval.

To address the challenges of noise stroke interference in the early stages of on-the-fly retrieval, we propose a novel noise-filtering framework for the on-the-fly SBIR problem. Our method mitigates the challenges identified above by evaluating the significance of user-drawn sketch strokes at both pixel and structural levels. Our framework consists of two key components: the stroke consistency detection module (SCDM) and adaptive gated mixture of experts module (AGMoE). The SCDM assesses the importance of sketch strokes from both pixel-level and contour structure perspectives, ensuring the accuracy of detection for both simple and complex strokes, and preserving the crucial sketch information required for model parsing and learning. The objective of the AGMoE module is to extract semantic information from filtered strokes comprehensively, aiming to avoid information loss and semantic leakage, thereby enabling the extraction of robust and discriminative features, even in the early stages.

To evaluate the generalization ability and robustness of the proposed model in handling complex strokes, we selected two categories of datasets: a set of simple sketch datasets (ChairV2 and ShoeV2) and detailed facial sketch dataset. Additionally, considering the diverse range of sketch styles encountered in practical applications, variations in user drawing habits, and the fact that existing datasets may not fully reflect real-world scenarios, we constructed a more challenging dataset based on the SKSF-A dataset [Yun et al., 2024], encompassing seven distinct sketch styles. The strokes in the sketch sequences of this dataset closely approximate the actual user drawing process. Model performance is evaluated from both qualitative and quantitative perspectives. The experimental results demonstrate that the proposed model exhibits excellent retrieval performance in the early stages of sketch drawing and is suitable for handling sketches of different styles. Our contributions are summarized as follows:

• (i) We propose a novel on-the-fly noise-filtering crossmodal retrieval framework. This framework effectively addresses the noise strokes generated during the drawing and retrieval process, thereby enhancing retrieval performance and user experience.
• (ii) We design a stroke evaluation and feature extraction mechanism integrating the SCDM and AGMoE module to identify key sketch information by assessing stroke importance at the pixel and contour levels, ensuring the maximal preservation of crucial semantic information while removing noise.

(iii) We construct a challenging multi-style sketch dataset that aligns closely with practical application scenarios. This dataset serves as a benchmark for evaluate the generalization ability and robustness of models in practical settings, providing a more relevant evaluation tool for future research.

2 RelatedWork

Sketch-Based Image Retrieval. SBIR has witnessed significant advancements, particularly in the fine-grained SBIR (FG-SBIR), which matches sketches to specific image instances. [Guo et al., 2017; Bhunia et al., 2021a; Bhunia et al., 2021b; Bhunia et al., 2022b; Zuo et al., 2024; Zhou et al., 2024; He et al., 2025]. Researchers have explored various strategies to enhance the performance of FG-SBIR, including leveraging deep Siamese triplet networks [Yu et al., 2016] and further optimize them by with attention-based higher-order loss functions and text tags [Bhunia et al., 2023; Fang et al., 2024]. While some studies address noisy sketches and style variations[Sain et al., 2021; Bandyopadhyay et al., 2024], key challenges like sketch style modeling in dynamic scenes and effective noise discrimination in intricate sketches still need further investigation.

Mixture-of-Experts. The MoE model uses a gating network to activate multiple expert subnetworks for task-specific processing. Its ability to manage data diversity has made it prominent in large language models (LLMs) [Carion et al., 2020; Li et al., 2022; Radford et al., 2021] following the rise of vision transformers [Rao et al., 2022; Jain et al., 2023; He et al., 2024] in the visual domain (e.g., VMoE [Riquelme et al., 2021], LlMoE [Mustafa et al., 2022], and CuMo [Li et al., 2024]). Given the inherent stylistic diversity in sketches, the MoE model can assign different styles to specific experts for tailored learning, enabling targeted modeling of sketch features. This study introduces an MoE module to leverage its advantages for handling diverse sketch styles, aiming to improve model understanding and generation in complex or noisy scenarios. This approach allows for finer-grained stylistic capture and efficient resource utilization, enhancing complex brushstrokes processing performance.

3 ProblemDefinition

On-the-fly SBIR aims to retrieve a target image as rapidly as possible using the fewest strokes. The drawing process is modeled as an ordered sketch sequence S = {s1, . . . , st, . . . , sn}, where st is the tth stroke and n is the total stroke count. At time t (1 ≤ t ≤ n), the input sketch is st. Consider a sketch sequence space DS , where S ∈ DS . Given an image database DI = {I1, . . . , Im} containing m images, the retrieval model dynamically evaluates the degree of matching between st and each Ij ∈ DI . A matching function f(st, Ij) ∈ R is defined to represent similarity. The model generates a dynamic ranking list Rt, arranging images in descending order based on their match with st. At each time instance t of dynamic retrieval, the model evaluates the match between the current sketch st and each image Ij ∈ DI . A matching function f(st, Ij) that outputs a scalar similarity measure is then defined. Based on this matching function, the model ranks the images in the database, generates a dynamic ranking list Rt, and orders images by decreasing match score st. Our objective is to construct a dynamic retrieval model where the rank of I ∈ Rt improves rapidly as the user adds strokes. Ideally, a minimum stroke count topt ≤ n exists such that for t ≥ topt, the target image I achieves the top rank, meaning Rank(I,Rt) = 1. We aim to retrieve the target image as quickly as possible, which translates to the minimization of topt. We attempt to optimize the matching function and ranking such that:

where ttop ∈ [1, n], and topt(f) denotes the minimum number of strokes under the matching function.

4 ProposedMethodology

Overview. We design a cross-modal, real-time retrieval system for noise filtering (See Fig. 2), comprising text and image branches for diverse query inputs. The text branch uses a text encoder and linear layer to reduce dimensionality, facilitating semantic matching with target images. The image branch handles images and sketches. For sketch queries, processing involves an image encoder, SCDM, and AGMoE module. Image queries use an image encoder and AGMoE. Dimensionally reduced sketch features are then fine-grained matched with corresponding image features. Text assists sketch matching, particularly for sparse early sketches, to mitigate retrieval randomness. Dimensionally reduced text and sketch features are fused into (v℘ st = fusion[vs i : vt ]) for image matching. Specifically, we present three salient designs. (i) A novel SCDM analyzes sketch stroke weights at two complementary levels, improving noise filtering reliability. Simultaneously, AGMoE maximizes semantic information retention post-denoising. (ii) Textual data assists retrieval of early, sparse sketches, effectively mitigating initial retrieval randomness. (iii) Training integrates unimodal contrastive and auxiliary fusion losses for fine-grained matching between combined queries and target images.

4.1 Stroke Consistency DetectionModule

The SCDM evaluates each stroke's importance from two complementary perspectives. First, we focus on pixel-level similarity between sketch strokes and target images. This direct comparison captures fine-grained correspondences to assess the contribution to detail representation. An image encoder EI(·) extracts embeddings V s i = EI(Si) for the current sketch stroke si and V p I = EI(T p ) for the target image T p . Pixel-level consistency is then measured using the cosine similarity between these embeddings.

Second, to enhance the perception of global structural information and assess stroke importance, we compare the current sketch with the target image's skeleton contour, Sbone. This aids in identifying whether strokes capture the target image's key structures. We calculate the cosine similarity between the embedding representations of the stroke and Sbone = K(T p ), described as Vsbone = EI(Sbone), using an image encoder to measure structural consistency. Combining pixel- and structural-level similarities yields a comprehensive stroke importance score, sim.

Where, w1 and w2 are dynamically adjusted parameters. For simple line drawings, precise pixel-level matching is less critical, so w is dynamically adjusted based on the sketched object's complexity.

To transform the continuous similarity score sim into discrete stroke weights for differentiated processing. we use the Gumbel-Softmax function. This function differentiably samples from a categorical distribution, yielding a discrete, sparse weight distribution. Specifically, Gumbel-Softmax generates a corresponding weight value for each stroke based on its comprehensive similarity score. Strokes with higher similarity scores receive higher weights, indicating their crucial role in representing the target image, while strokes with lower similarity scores receive lower weights, suggesting they may be noise or redundant information.

4.2 Adaptive GatingMoEModule

We use MoE to better capture diverse sketch strokes for enhanced quality. To address MoE's limitations, we propose the AGMoE module, which improves model robustness through a Top-k gating mechanism and adaptive noise.

As shown in Fig. 2, the denoised sketch strokes V s′ i serve as the inputs for the AGMoE module, which primarily consists of N expert networks Experti : RD → RD , i = 1, 2, 3, . . . , n, a gating network G : RDgate → RN , and parameters wnoise ∈ RD×N , which are utilized for jointly adjusting noise. First, the input stroke Xinput passes through the gating network G(·) to generate logit values (llogits) for each expert network, which are used to measure the degree of applicability of each expert to the current input stroke.

Where, B, T , and N denote the batch size, stroke length, and number of expert networks, respectively. During training, we add adaptive noise to the gating network's output logits (llogits) to enhance the module's exploration capabilities and increase expert selection diversity. This randomness encourages the model to explore various expert combinations during training, improving its generalization ability. When the training condition (train = True) is satisfied and noisy gating is enabled (noisygating = True) , the logits are dynamically adjusted by the following formula:

Where α = 10-2 is a constant ensuring the noise's standard deviation is always greater than zero. The softplus function correlates noise variance with the magnitude of llogits achieving adaptive noise injection. Depending on whether noisy gating is enabled during the training phase, the final l′logits are calculated as follows:

Subsequently, to enhance computational efficiency and direct the model's attention to the most relevant experts, we introduce the Top-k gating mechanism. To ensure that softmax normalization is computed only on the selected Top-k logits, we set the values of the unselected logits to negative infinity, resulting in a sparse vector lsparse. By applying softmax normalization to lsparse, we obtain the gating weights G(·) representing the importance of each selected expert.

Each expert network is a simple multilayer perceptron with two linear layers and a rectified linear unit activation function.

For each selected Experti, i ∈ {1, 2, 3, ..., k}, its output for input ˜x is computed as Yi = Experti(˜x) ∈ RB×T×D . The outputs of all k selected experts are then stacked.

Finally, we apply the gating weights to weight the output of each expert, thereby obtaining the final weighted output.

Where, ⊗ denotes element-wise multiplication. The gating network scales each expert's output based on its corresponding importance.

4.3 Expert Contrastive Learning

This section outlines our model's training, which incorporates two auxiliary tasks designed for noisy stroke filtering during interactive drawing.

(1) Cross-modal alignment. To account for input modality variations, specialized alignment tasks are performed by different experts on specific modalities. We design a crossmodal alignment learning task to explore the specialized knowledge and skills of each expert for specific alignment tasks, thereby facilitating fine-grained learning and optimization. Image-text contrastive learning (ITC loss [Radford et al., 2021]) ensures close alignment of text and images within the feature space, optimizing their semantic consistency. Sketch image contrastive learning (ISC loss [Radford et al., 2021]) maps sketches and their corresponding facial images to a shared feature space.

(2) Bimodal alignment. Leveraging the complementarity of multimodal information further enhances the robustness of the model. When one modality's information is missing or excessively noisy, the model can rely on the other, preventing significant overall performance impact. ITSC loss calculates fused features as follows:

Where, v℘ st is the embedding vector after fusing sketch and text, and τ is a temperature parameter.

5 Experiments

Datasets. To select appropriate datasets based on sketch stroke complexity, we use eight publicly available datasets (See Table 1) [Song et al., 2018; Muhammad et al., 2018; Pang et al., 2019; Bhunia et al., 2020; Dai et al., 2022; Liu et al., 2024; Dai et al., 2024b]. Furthermore, to test our model's generalizability across diverse styles, we expand the SKSF-A dataset [Yun et al., 2024], which encompasses seven sketch styles, to simulate scenarios with the stylistic diversity and cluttered strokes encountered in practical applications.

Implementation Details. The model is trained using RTX4090 GPUs. Batch sizes are eight for simple stroke training and 16 for facial sketch training. The model trains for 50 epochs on eight public datasets using the AdamW optimizer with a 0.05 weight decay factor. The AGMoE module employs eight expert networks with a top-k value of four. A cosine-annealing learning rate schedule is adopted, with the learning rate varying between 3e-5 and zero. During training, the ChairV2 and ShoeV2 datasets have 20 total sketch strokes and 64 output dimensions. The total number of strokes in the user-drawn dataset varied.

Evaluation Metrics. (i) We employ the standard FG-SBIR evaluation metric, Acc.@q, which quantifies the percentage of sketches where the true paired photograph appears within the top-q retrieval results. (ii) On-the-fly: Given our prioritization of the target images appearing at the top of the retrieval lists, we select the metrics m@A (rank percentile) and m@B (percentage of sketches with 1/rank) [Bhunia et al., 2020] to gauge average retrieval performance across all sketching stages. To evaluate early-stage sketching performance, we use w@MA and w@MB. Higher values of these metrics indicate better performance.

Competitors. TriNN-C [Yu et al., 2016] and TriNN-SR [Bhunia et al., 2020] use a Sketch-a-Net backbone with standard triplet loss. TriNN-RL( TS) [Bhunia et al., 2020] uses an on-the-fly retrieval framework based on representation learning (RL). LSTMB [Liu et al., 2022] is a Bi-LSTMmodule optimizing partial sketch sequences. PSERL/N [Liu et al., 2024] optimizes sketch representations using prior knowledge. We compare the model without labels. Cross-Hier [Sain et al., 2020] utilizes a cross-modal hierarchy with expensive paired embedding. StyleMeUp [Sain et al., 2021] utilizes model-agnostic meta learning training. NSR [Koley et al., 2024a] is a lightweight, portable, and interpretable seamless plugin. Seql [Dai et al., 2023] is a sequential learning method that integrates convolutional neural network and lstm modules. MGAL [Dai et al., 2022] and MGRL [Wang et al., 2025] are based on multi-grained RL, using Sketch-a-Net as backbone. MITRL [Dai et al., 2024b] is a multimodal retrieval model trained using LLMs. We also compare its performance against CLIP [Radford et al., 2021], BLIP [Li et al., 2022] and FVIP [Dai et al., 2024a].

5.1 Comparisons With SOTAResults

To evaluate model performance in terms of identifying noisy strokes, we conduct experiments on simple sketch and face sketch datasets. The results are detailed below.

Performance Analysis on Simple Lines. Given the inherent lack of textual information in the ChairV2 and ShoeV2 datasets, the text branch's input prompt during model training is set to "a photo of f'{color} chair/shoe ' ". We chose color as the textual prompt due to sketches' inherent blackand-white characteristics.

(1) Partial/Early Retrieval. As illustrated in Fig. 3, the retrieval performance of all methods exhibits an upward trend as the sketch is progressively completed and tends to stabilize when the sketch is near completion. Compared with other models, our model demonstrates a significant advantage in the initial stages of sketch drawing, as it can rapidly and accurately retrieve the target image with very few strokes. These results indicate that the proposed method can effectively identify keystrokes in the early stages of sketching and overcome potential interference strokes during the drawing process. To address the challenges of diverse styles, Fig. 4 presents a retrieval comparison of two groups of sketches. Our model exhibits the ability to maintain efficient retrieval performance in the presence of such variations.

(2) Complete sketch Retrieval. As Table 2 shows, our performed excellently on datasets. On the ChairV2 dataset, our method demonstrated significant improvements compared with most existing methods. In terms of Acc.@5, the performance of our model is slightly lower than that of the StyleMeUp model but still highly competitive. On the ShoeV2 dataset, our model's advantages are more significant, showing notable improvements over the best baselines. This is likely because ChairV2 has higher similarity than ShoeV2. When considering the results comprehensively, our model outperform all other models.

Performance Analysis on Facial Sketches. Table 3 compares our model with previous models. Our method clearly outperforms others in the early retrieval stage. Notably, our method achieves significant performance gains over baseline models trained with triplet loss. This is because in the initial stages of sketching, as a result of the sparsity of strokes and frequent presence of noise, simpler methods (e.g., reducing sketch-positive sample distance) are susceptible to noise, potentially misleading the model's training direction. Additionally, incorporating textual information substantially improves retrieval performance. This improvement stems from textual information's ability to mitigate inherent randomness caused by limited sketch details. After training our approach with the FVIP language model, we observed a marked enhancement in its performance. The results demonstrate that effectively identifying interfering strokes during the drawing process and accurately matching themwith the target image is crucial for improving the model's early retrieval performance.

5.2 Diverse Sketch Style Analysis

Sketch drawing styles vary significantly between individuals, with more detailed facial sketches showing even greater diversity due to increased complexity. To investigate our model's ability to manage multiple styles effectively, we extended the SKSF-A dataset to include seven distinct sketch styles, as shown in Fig.5. We present seven distinct stylistic sketches and demonstrate the retrieval process for each. The results indicate that retrieval performance exhibits significant fluctuations across different styles. Therefore, maintaining the model's robustness against stroke noise and stylistic variations is crucial for ensuring retrieval stability.

Table 4 presents the model's retrieval performance across different sketch styles. Notably, the m@A metric maintained excellent level across all styles, generally exceeding 90%. This demonstrates the model's effective recognition capability for sketches of varying styles and its excellent ability to rank candidate results. However, we observe the model's m@B value is lowest for Style6 and peaked for Style5, exhibiting a significant 65.79% difference. The model's performance in terms of the Acc.@1 and Acc.@5 metrics aligns with the trend observed for the m@A, with Acc.@5 showing robust performance across all styles. When comparing early retrieval performance with complete sketch retrieval, Style6 performed poorest in both m@B and Acc.@1. These results indicate that sketches in style6 may possess unique visual characteristics that pose a challenge to the model's precise identification. In summary, the experimental results reveal inherent performance differences in the model's handling of different sketch styles, underscoring the importance of considering sketch style during model training to mitigate its impact on recognition performance.

5.3 Component Ablation Analysis

To evaluate the contribution of each component, we conducted ablation experiments on four datasets. For the ChairV2 datasets, we employed the CLIP model for weight initialization, whereas facial sketches are pre-trained using FVIP. The AGMoE( N) sub-table represents the presence or absence of added noise in the AGMoE modules. Ours-full represents the training configuration using all components.

The results in Table 5 indicate that the model incorporating the SCDM outperforms the AGMoE module model without the SCDM or added noise. It can be concluded that the SCDM enhances the recognition accuracy of the model, although its impact varied slightly across different metrics. For the ChairV2 datasets, models utilizing the SCDM consistently exhibited improvements across all metrics, regardless of whether noise is added. This result indicates that the MoE architecture can effectively capture and process diverse sketch strokes, ensuring a high degree of consistency in stroke quality and style. These ablation experiments confirm the positive impact of both the SCDM and AGMoE module on model performance, with their combination yielding optimal results. Across all four datasets, the proposed method demonstrated superior performance, validating the effectiveness of our framework.

5.4 Analysis Under Practical Applications

Significant variations in user drawing habits, including skill, style, and stroke length, contribute to increased stroke complexity. To simulate realistic sketching scenarios, we use three datasets with diverse styles created by numerous volunteers. Distinct sketch styles are presented in Fig. 6. Evidently, sketching styles immediately exhibit noticeable differences. However, our method demonstrates the ability to retrieve target images accurately, even with a limited number of strokes.

As shown in in Table 6, the retrieval performance of all compared methods declin across the datasets, indicating that both sketch style and stroke order influence retrieval effectiveness. Regardless of whether the sketches are simple line drawings or complex facial sketches, our method exhibited superior retrieval performance on diverse and complex datasets. This outstanding performance can be attributed to the effective identification of early noisy strokes, ensuring semantic consistency throughout the sketching process and maintaining strong early retrieval performance.

6 Conclusion

We propose a novel on-the-fly noise filtering cross-modal retrieval framework designed to address the challenges of noisy stroke interference and effective feature extraction in the early stages of instant retrieval. Extensive experimental results thoroughly validate that our proposed method can effectively identify noisy strokes in various sketches, including simple line drawings and complex sketches, thereby significantly improving the initial retrieval performance. Furthermore, we have construct a multi-style dataset to comprehensively evaluate the generalization ability and robustness of the model under complex stroke scenarios.

Acknowledgements

This work is sponsored by National Nature Science Foundation of China (No.62221005 and U2336212), Nature Science Foundation of Chongqing (No.2023NSCQ-MSX0054), Intelligent Policing Key Laboratory of Sichuan Province (No.ZNJW2025KFMS005) and Doctor Student Innovative Talent Program of Chongqing University of Posts and Telecommunications (No.BYJS202405). Dawei Dai is the corresponding author.