Self-Supervised Learning for Enhancing Spatial Awareness in Free-Hand Sketches

1 Introduction

Free-hand sketch is a versatile medium of communication, conveying subjective impressions of the objective world and serving as a vehicle for emotional expression [Xu et al., 2022]. It is a unique visual form which can be viewed as a collection of strokes arranged in a particular spatial layout, including relative positions, orientations, inclusions, and more. The spatial layout of strokes is an important factor, because even minor changes in stroke position may make them difficult to recognize. Due to time constraints and the manual nature of hand drawing [Ha and Eck, 2018], sketches can be variable, abstract, and iconic, making it a challenge to learn robust and accurate sketch representations.

∗Corresponding author

Recently, graphic sketch representations [Qi et al., 2021; Su et al., 2020; Zang et al., 2023] have been proven in representing sketches effectively. These methods use local visual cues of a sketch as nodes and connect the nodes based on temporal proximity [Su et al., 2020], spatial proximity [Qi et al., 2021], or semantic proximity [Zang et al., 2023] to construct the corresponding graph. Finally, a Graph Convolutional Network (GCN) [Kipf and Welling, 2016] is used to aggregate information from graph nodes. But even minor changes in the spatial layout of strokes can significantly affect these local visual cues.

However, existing methods overlook the significance of the spatial layout of strokes. On one hand, strokes being drawn in the wrong positions is a common phenomenon. Factors such as casual sketching, time constraints, and physical conditions like micrographia [Armstrong and Okun, 2020] in patients with Parkinson's disease can contribute to these unavoidable placement errors, hindering the comprehension of the sketch by humans or models. On the other hand, in contrast to the spatial layout, visual cues in local regions are susceptible to being disrupted by these placement errors. As shown by the variations in region A in Figure 1(a) and the small changes in the spatial layout of regions B, C, and D in Figure 1(b). Existing methods have difficulties in handling these placement errors of strokes because they construct graph with a specific type of proximity, rather than the overall spatial layout. These issues impede the extraction of spatial layout information and the acquisition of robust sketch representations.

To investigate the relation between the spatial layout of strokes and the acquisition of sketch representations, this paper designs a self-supervised task, called Sketch Reorganization, aimed at rectifying the misplacement of strokes which is common in everyday life. Proposing and addressing this task can aid in humans sketching and understanding sketches. Similarly, it can help acquire more robust sketch representations. Unlike conventional sketch restoration tasks [Zhao et al., 2019; Bhunia et al., 2021; Su et al., 2020; Qi et al., 2022], our task, for the first time, takes into account human-introduced errors at the vector level, which aligns with the sequential nature of sketches. Unfortunately, existing methods face challenges in accomplishing this task due to a lack of overall spatial layout awareness capability, which can alleviate the impact of placement errors.

In this paper, we propose SketchGloc: a spatially aware method with a multi-graphs for graphic sketch representations. To measure the spatial layout between strokes of varying sizes and shapes, we took inspiration from the object detection field's practice [Zou et al., 2023; Redmon et al., 2016; Liu et al., 2016; Girshick, 2015] of using grids and bounding boxes. The geometric localization module of SketchGloc constructs the spatial layout between strokes as multi-graphs using the grids. Specifically, the strokes are encoded as the nodes and the process of constructing the multiple graphs is as follows. We began by drawing different grids for each stroke. (e.g. red and blue gridlines in Figure 1(c) representing two different grids and strokes). Then, one stroke is placed on the grids of another stroke, which changes the state of occupied grid. (e.g. the colors of the cells occupied by the strokes in Figure 1(c) differ). Finally, two strokes will be linked on a specific graph based on whether a particular cell is occupied. (e.g. in Figure 1(c),two strokes(Above) are linked on the 'top-right' graph). Our method is particularly well-suited for sketches, as using grids to represent spatial layout helps alleviate the impact of placement errors on the spatial layout. Additionally, the final sketch representation is synthesized from multiple graphs, which facilitates learning of the overall spatial information of the sketch and produces an accurate representation.

The contributions of the paper can be summarized as follows:

• We design a self-supervised learning task 'Sketch Reor-
• ganization' that takes the errors of stroke placement into account. This task helps to improve the robustness of the sketch representation.
• We propose SketchGloc, which constructs multi-graphs based on the spatial layout between strokes. By using grids to represent spatial layout, the impact of stroke placement errors is alleviated. This method helps to learn the overall spatial layout information and produces more accurate sketch representations.
• The experimental results show that SketchGloc significantly improves the robustness of represenations in controllable sketch synthesis task and outperformes other methods in the Sketch Reorganization task. Additionally, the experiments show the practical potential of SketchGloc in aiding humans to draw and comprehend sketches.

2 RelatedWork

2.1 Sketch Representation Learning

The field of sketch generation [Ha and Eck, 2017; Zang et al., 2021; Vinker et al., 2022; Vinker et al., 2023; Voynov et al., 2023; Li et al., 2024] has been a subject of extensive research, as sketches can be represented in various ways. Notably, the groundbreaking research conducted by sketch-rnn [Ha and Eck, 2017] has been highly recognized. Its significant contributions have captured the attention of scholars and researchers. This method falls into the encoder decoder paradigm and consists of two Long Short-Term Memory (LSTM). Additionally, the pixel-based sketch encoders [Chen et al., 2017; Zang et al., 2021; Xu et al., 2020a] have received significant performance improvement due to their remarkable spatial feature extraction ability. It is noteworthy that generative GAN-based [Ge et al., 2020] and Diffusion-based models [Xing et al., 2024; Qu et al., 2023; Wang et al., 2022] are effective for capturing the semantic properties of sketch representations due to their excellent generative performance.

In recent times, Graph Convolutional Network (GCN) based sketch encoders [Qi et al., 2021; Qi et al., 2022; Zang et al., 2023] have emerged as a promising research method, capitalizing on the advantages of integrating temporal and spatial features. SketchLattice [Qi et al., 2021] proposed a method for vector sketch representation, with latticed coordinates [Pan et al., 2020] sampling as nodes, and edges connecting nodes based on similarity in the hidden space. In an effort to exploit the temporal characteristics of sketches, another approach was introduced by [Qi et al., 2022], which incorporates both vector and pixel formats. This method sequentially samples sketches and employs features extracted by a Convolutional Neural Network (CNN) as graph nodes in the order of sampling. Furthermore, the semantic properties of node connectivity were explored by [Zang et al., 2023], building upon the node construction method, resulting in improved experimental performance.

2.2 Self-Supervised Pretext Tasks

Self-supervised learning [Gidaris et al., 2018] has achieved remarkable success in the field of images, excelling in various tasks such as classification and semantic segmentation. Defining an appropriate pre-text task is crucial in selfsupervised learning. In the field of image analysis, prevalent pretext tasks include predicting relative positions [Doersch et al., 2015], image completion [Doersch et al., 2015], image coloring [Pathak et al., 2016], and deformation prediction [Doersch et al., 2015], among others. In the field of sketch, there are also research efforts on pre-text tasks for self-supervised learning. These tasks differ from those in the image domain, as they must consider the unique data characteristics [Xu et al., 2020b] of sketch.

Existing tasks for self-supervised learning of sketches are akin to those in the image field, primarily focusing on visual feature prediction or complementation [Xu et al., 2020b; Su et al., 2020]. While these methods are tailored for sketch data format, their primary focus remains on leveraging vector features to aid model learning, specifically targeting the prediction of missing visual features.

3 Methodology

Figure 2 provides an overview of the SketchGloc method, which is utilized to achieve robust learning of representations for vector sketches and to address our proposed selfsupervised task (Sketch Reorganization) specific to sketch vector features. Initially, we consider the stroke s in a sketch S as the fundamental unit for model input. A sketch S is represented as a sequence of points (∆x, ∆y, p1, p2, p3), where (p1, p2, p3) denotes the state information at the time the sketch is drawn. The values (1, 0, 0), (0, 1, 0), and (0, 0, 1) correspond to touch, lift, and end of pen, respectively. We use the lifted state (0, 1, 0) during a single drawing as a marker for transition from the end of the previous stroke sm-1 to the beginning of the new stroke sm. The processed sketch with M strokes is represented as S = {sm|1 ≤ m ≤ M}, where each stroke has at most N points. For each stroke sm, we feed it into the LSTM, as it has been widely used in previous works , encoder qϕ to obtain representations vm(1 ≤ m ≤ M) as graph nodes. To establish edges between graph nodes vm, we utilize the Geometric Positioning Module F to compute the spatial layout coding D between strokes and subsequently transform it into nine adjacency matrices A, which depends on the grids settings in this paper. This process enables GCN to effectively capture the spatial layout information between strokes. Finally, the GCN encoder qξ calculates the final code y, which is utilized by the LSTM decoder pθ to generate the sketch ˜ S.

3.1 Linking Strokes by Geometric Localization Module

As is well known, the spatial layout between two strokes (sm1 , sm2 ) can be straightforwardly described by the grids. To address more intricate stroke spatial layout relationships, we leverage multiple spatial layout information. In this paper, a nine-bit spatial layout encoding D = {di|1 ≤ i ≤ 9} is introduced to represent nine distinct spatial layout information. We formalize it:

where F : RN×5 × RN×5 → F M×M×9 2 . Each element di ∈ D(m1,m2) indicates whether sm2 is in the ith cell of the grids of sm1 .

To effectively integrate information in the graph network qξ based on the spatial layout between strokes, we split the spatial layout encoding matrix D into nine different graph adjacency matrices {A(i) |1 ≤ i ≤ 9}. Each adjacency matrix A(i) represents one bit of spatial information for each pair of strokes. At the output layer of the graph network qξ, a post-fusion strategy will be implemented to combine the results of message passing in each spatial layout. Formally, we describe this process as follows:

For a specific graph adjacency matrix A(i) , we impose a constraint that restricts the propagation of message among graph nodes to occur exclusively in the same spatial layout, thereby appropriately adjusting its significance. Furthermore, each adjacency matrix is self-connected, as indicated by a(i) k,k = 1.

The spatial layout of strokes obtained through grids is in a discrete form, which aligns well with the construction requirements of graphs. Grids precision can be controlled by adjusting the number of grid cells, such as setting it to 5 × 5. However, free-hand sketches are inherently sparse. Therefore, we choose for a 3×3 grid, which is deemed suitable for the current task.

3.2 Graph Networks Based on Stroke-Level Sketches

We utilized spatial layout encoding to construct the sketch GCN Encoder q(i) ξ (y|V ,A(i) ). To comprehensively utilize all spatial layout information in the spatial layout encoding D, we employ nine distinct GCN layers q(i) ξ ∈ qξ to integrate the information from graph nodes in each graph. Each GCN layer q(i) ξ is constructed for graph nodes V = {v1, v2, ..., vM} with adjacency matrices A(i) to compute the propagation results h(i) of sketch strokes in the ith grid. The weight of each GCN layer q(i) ξ is denoted byW (i) :

This final feature h is then passed through our fully connected layer to yield two vectors µ and σ. These vectors are computed using the formula y = µ + σ × ε, where ε represents noise sampled from the Gaussian distribution G(0, 1). The resulting vectors µ and σ constitute the final code y for the sketch ˜ S.

3.3 Disrupt Vector Sketch

To achieve Sketch Reorganization task, we require a method for creating a sketch with placement errors based on a complete sketch. The random noise ϵ follows a Gaussian distribution G(ϵ|0, 1). To control the impact of the noise ϵ and maintain the disruption of the sketch at a reasonable level, we introduce an additional parameter (scale = 10%, 30% ). The scale represents a proportion of the sketch size of S, and its

Algorithm 1: Disturbing sketch strokes

Data: S = s1, s2, ..., sM, scale Result: S 1 Initialization:i ← 0 2 Calculate the size of the sketch S // add noise 3 while i ̸= n do 4 S[i,0], S[i,1] ← xabs +S[i,0], yabs +S[i,1] // Adding noise to each of the strokes 5 if i ̸= 0 then 6 ϵx, ϵy ∼ G(0, 1), G(0, 1) 7 Add noise ϵx × size × scale, ϵy × size × scale to si 8 end 9 i ← i + 1 10 end

product with the size of the sketch is used as a multiplier for Gaussian noise ϵ. We accomplish this step by introducing random noise ϵ ∼ G(ϵ|0, 1) to strokes positions.

3.4 Training a SketchGloc via Sketch Reconstruction

We pass the final code y of the sketch S to a Recurrent Neural Network(RNN) decoder pθ(S|y), which sequentially reconstructs the sketch ˜ S. Our training objective is defined as follows:

The first likelihood function in the objective necessitates reconstructing the correct input in sequence format. Similarly, for sketch S which is disrupted, we input the decoder with its final code y, and the RNN decoder pθ will generate the complete sketch in sequence format.

4 Experiments

We selected controllable sketch synthesis [Zang et al., 2021] and the Sketch Reorganization task proposed by us to validate whether SketchGloc has learned accurate and robust graphic sketch representations.

4.1 Preparation

Datasets. We evaluated SketchGloc onQuickDraw [Ha and Eck, 2018], a large vector sketch dataset containing tens of millions of human free-hand sketches across 345 classes. To account for variability between different sketch classes, we utilized three datasets proposed by [Zang et al., 2021] in our experiments. DS1 and DS2 were taken from [Zang et al., 2021], with DS1 including sketches of bee, word, giraffe, bus, and pig, allowing us to assess sketch generation with large gaps between classes. DS2 encompassed sketches of airplane, angel, apple, butterfly, bus, cake, fish, spider, the Great Wall, and umbrella, presenting more diverse categorical patterns to evaluate the model's ability to distinguish across multiple categories. DS3 [Qi et al., 2021] includes car, cat and horse additionally to DS1, constituting a more challenging data set. For each category, we used 70,000 sketches for training, 2,500 for validation, and 2,500 for testing.

Baselines. Considering the encoder classification [Xu et al., 2022; Li et al., 2020; Lin et al., 2020; Ribeiro et al., 2020] of sketch generation models, we selected six models as baselines for our experiments. The sketch-rnn [Ha and Eck, 2017] model, widely used as a benchmark for tasks involving vector sketch generation, is the most representative model for processing sketches in sequence format. Additionally, sketchpix2seq [Chen et al., 2017] and RPCL-pix2seq [Zang et al., 2021] take images as input and leverage local structure concept to enhance performance. Moreover, RPCL-pix2seq further imposes constraints on the latent space. Song et al. [Song et al., 2018] model processes both sequences and images, treating each sequence-image pair as a joint input. SketchHealer [Su et al., 2020] extracts information from CNN and utilizes a graph network to derive a sketch representation. SketchLattice [Qi et al., 2021] constructs a node representation from point-only Euclidean distance within a latticed coordinates network to establish node edges. The semantic properties of node connectivity were explored by [Zang et al., 2023], building upon the node construction method, resulting in improved experimental performance. The models were trained using the officially published source code.

We set the number of strokes M, and the batch size N, to 50 and 128, respectively. The Adam optimizer was employed for learning with parameters β1 = 0.9 and β2 = 0.999. Additional experimental details are provided in the appendix.

4.2 Controllable Sketch Synthesis

Controllable sketch synthesis involves generating accurate input sketches, Model's performance is evaluated using the metrics Rec and Ret, as proposed in [Zang et al., 2021]. Rec indicates whether the generated sketch ˜ S belongs to the same class as its input sketch S. For performance testing, we trained three sketch-a-net [Yu et al., 2017] models on three datasets (DS1,DS2,DS3). Additionally, Ret measures whether the input sketch S and the generated sketch ˜ S are encoded as close as possible in the latent space. To achieve this, we fed the output ˜ S of the network, along with the original input S, into the encoder to obtain the latent variables ˜ y vs. y. The original y values obtained using the entire test set were used to retrieve ˜ y. Ret represents the success rate of the retrieval process, as shown in Figure 5.

Table 1 reports the controllable sketch synthesis performance. SketchGloc considers the use of successive strokes of a sketch as graph nodes and incorporates multiple graphs based on the spatial layout relationships of the strokes, making it better suited for generating vector sketches. By directly encoding the stroke information and passing the sketch over the graph space using a geometric localization adjacency matrix, the method can accurately generate sketches into the latent code. Our approach enables the simultaneous existence of nine graph structures based on different geometric information,leading to accurate sketch representation. As a result, our method is comparable to current state-of-the-art methods.

4.3 Sketch Reorganization

Sketch Reorganization is a task aimed at reorganizing a complete original sketch S using a disturbed sketch S′ . The success of Sketch Reorganization is determined by how similar the generated sketch ˜ S is to the original sketch S. To evaluate the performance of the models in this task, we use metrics Rec and Ret [Zang et al., 2021], similar to those used in controllable sketch synthesis. These metrics help us answer two key questions: (1) After Sketch Reorganization, how distinguishable are the outputs of different models? (2) Are the latent variable codes stable across models?

As shown in the Table 2, our method outperforms the other methods in Sketch Reorganization. SketchGloc successfully alleviate the impact of placement errors on the spatial layout, carefully considering the possibility of errors in graph node connections. We employ geometric localization module, a suitable module for obtaining spatial information for sketches, enabling accurate learning of sketch graph representations. Table 2 also reports the metrics of other methods. sketch-rnn [Ha and Eck, 2017] requires high accuracy in sequence data, leading to poor performance in the sketch representation when errors occur in the strokes, hindering proper reasoning about the semantics of the original sketch. sketch-pix2seq [Chen et al., 2017] and SketchLattice [Qi et al., 2021] methods, sensitive to the visual structure of the sketch, produce inaccurate representations when even small disruptions in the strokes cause significant changes in pixel space. SP-gra2seq [Zang et al., 2023] exhibits performance improvements at scale of 10%, leveraging local graph nodes, but struggles to reason effectively about overall sketch semantics at scale = 30%.

Additionally, we provide a qualitative comparison in the Figure 3, showing that SketchGloc achieves nearly perfect Sketch Reorganization at scale = 10%, resulting in plausible and comparable representations to the original sketch. In contrast, other methods do not reason effectively for most examples, especially at scale = 30%, where all methods except ours fail almost completely. SketchGloc still provides satisfactory results for most examples.

Certainly, it is worth mentioning that our method not only accomplishes the Sketch Reorganization task but also possesses the capability to optimize sketches. This is due to the assumption of errors in sketch inputs for the reorganization task. As shown in the Figure 4, quality of recreated sketches are improved. The generated sketches show a consistent and significant improvement in quality, contributing to an improved ability for individuals to both create and understand sketches. This advancement can also optimize the quality of sketches from impaired drawers, such as individuals with Parkinson's disease [Armstrong and Okun, 2020], assisting in communication or restoring cognitive abilities [Chancellor et al., 2014]. Readers can also optimize their understanding when faced with low-quality sketches.

4.4 Ablation Study

Performance Gained From Sketch Reorganization Task. As shown in the Table 3, under the Sketch Reorganization task settings(SR = yes), representations learned are more robust, achieving higher Ret performance in most cases. This is because, in the context of this task, sketch representations learn more robust features, making it less prone to confusion with similar sketches. This indicates the rationality of applying this task to sketch datasets.

Performance Gained From Geometric Localization Module. In order to investigate whether the introduction of the spatial geometric localization module in SketchGloc facilitates the establishment of relationships between stroke nodes, we conduct a comparative analysis in this section. We contrast Sketch Reorganization outcomes when employing a randomly linked sketch graph. Our experimentation involves calculating results at scale = 10% and scale = 30%. As presented in Table 4, we observed that the randomly connected graph, achieved by altering the adjacency matrix A in Eq.(4), yields inferior Sketch Reorganization performance at scale = 10% compared to our proposed method. Furthermore, this adverse impact becomes particularly pronounced when the sketch encounters a greater degree of disruption.

5 Conclusion

This paper introduces the Sketch Reorganization task, which aims to guide machines in correcting placement errors of strokes. To address this issue, we propose the SketchGloc method. This method utilizes spatial layout information to construct multiple graphs, enabling overall spatial layout information and enhancing the accuracy of sketch representations. By using grids to represent spatial layout can accommodate placement errors, the robustness of representation is improved. The effectiveness of SketchGloc has been demonstrated through extensive experiments involving controllable sketch synthesis and Sketch Reorganization.

Acknowledgements

This work was supported by the Shanghai Municipal Science and Technology Major Project, China (Grant No. 2021SHZDZX0102), and by the National Natural Science Foundation of China (grants No. 62172273), and the Fundamental Research Funds for the Central Universities (2232024D-28).