Rethinking Image Aesthetics Assessment: Models, Datasets and Benchmarks

1 Introduction

Can you tell which image in Fig. 1 is more beautiful? It is difficult to assess and compare the aesthetic perception of images with different themes because they correspond to different evaluation criteria, e.g., large and strong trees versus small and delicate flowers, all of which may be beautiful. Classic photography [Barnbaum, 2017] has shown that image aesthetics are strongly related to the theme; to understand the aesthetics of an image, photographers first pay attention to the theme of the image and then apply different evaluation criteria for images with different themes (image→theme→aesthetics).

∗Contact Author. This work was supported by the national key R & D program intergovernmental international science and technology innovation cooperation project (2021YFE0101600).

Utilizing several existing datasets (Table 1), previous studies directly learned aesthetics from images annotated with aesthetic scores. However, images are composed explicitly of natural signals; in other words, the elements based on pixels cannot contain any abstract or thematic information. The aesthetic scores given by the human subconscious are related to themes, but existing methods do not consider or take advantage of this. Thus, existing methods have attempted to build image→aesthetic mappings, which violate the process of human visual perception.

This mapping approach worsens image aesthetics assessment (IAA) in two respects: 1) The annotations of existing datasets do not consider that different themes have different scoring criteria; furthermore, all images are mixed together and scored without differentiation, which introduces considerable noise and error into the ground truth. 2) Even using the most advanced methods, learning directly from low-level pixels while using noisy ground truth as the supervision information makes it difficult to effectively perceive aesthetic information, and these methods are not powerful enough to understand the aesthetics, which causes attention dispersion (Fig. 2 and Fig. 3). These limitations inhibit the development of IAA, which is not yet fully understood.

To comprehensively investigate this topic, we provide two contributions. First, we build a large-scale dataset called the Theme and Aesthetics Dataset with 66K images (TAD66K), which is specifically designed for IAA. Specifically, 1) it is a theme-oriented dataset containing 66K images covering 47 popular themes. All images were carefully selected by hand based on the theme. 2) In addition to common aesthetic criteria, we provide 47 criteria for the 47 themes. Images of each theme are annotated independently, and each image contains at least 1200 effective annotations (so far the richest annotations). These high-quality annotations could help to provide deeper insight into the performance of models.

Second, we propose a baseline model, called the Theme and Aesthetics Network (TANet), which can maintain a constant perception of aesthetics to effectively deal with the problem of attention dispersion. Moreover, TANet can adaptively learn the rules for predicting aesthetics according to a recognized theme. To further improve the perception of each theme, we propose an RGB-distribution-aware attention network (RGBNet) to help the network perceive the color distribution in the RGB space and solve the problems associated with the high complexity of standard attention. Additionally, using TAD66K and two existing datasets, namely, Aesthetic Visual Analysis (AVA) [Murray et al., 2012] and Flickr Images with Aesthetics Annotation Dataset (FLICKR-AES) [Ren et al., 2017], which are widely used to verify general and personalized aesthetic models, we offer a rigid evaluation of 17 state-of-the-art (SOTA) baselines (Table 2), making this work the most complete IAA benchmark thus far. Promising results are achieved on all of the above benchmarks, clearly demonstrating the effectiveness of our model.

2 RelatedWork

IAA Datasets. Over the past few years, several datasets have been constructed for IAA (Table 1). The datasets DP Challenge [Datta et al., 2008] and Photo.Net [Joshi et al., 2011] were adopted early on, but they contain relatively few images, and these images comprise coarse annotations. Murray et al. [Murray et al., 2012] created AVA, a large-scale IAA dataset with nearly 255,000 images that has become one of the most popular IAA datasets. Subsequently, Kong et al. [Kong et al., 2016] provided a novel dataset called the Aesthetics and Attributes Database (AADB) that includes the aesthetic and attribute scores of multiple images by individual users. Yu et al. [Chang et al., 2017] and Jin et al. [Jin et al., 2019] compiled datasets called the Photo Critique Captioning Dataset (PCCD) and DPC-Captions, respectively, to solve the aesthetic-related photo caption generation problem. Ren et al. [Ren et al., 2017] created FLICKR-AES for personalized image aesthetics. CUHK-PQ [Luo et al., 2011] is the first dataset to organize images by theme, but it contains only 7 themes. More recently, Fang et al. [Fang et al., 2020] introduced the Smartphone Photography Attribute and Quality (SPAQ) dataset with 9 themes. Although the above datasets have advanced the field of IAA to various degrees, they fail to consider that the same evaluation criteria are not suitable for highly variable themes, which causes a label noise problem.

Unlike those of the existing datasets, the goal of TAD66K is to provide a more challenging and theme-oriented dataset. To the best of our knowledge, TAD66K is the largest densely annotated dataset (Table 1). Furthermore, TAD66K provides carefully selected themes, targeted evaluation criteria, and measures to prevent the long-tailed effect that occurs in the existing datasets; these features make TAD66K a solid and high-quality dataset.

IAA Models. The early models described in Table 2 focused mainly on extracting aesthetic information from im-

Table 2: Summary of 17 existing representative IAA models and the proposed TANet. Training Set: We count only datasets related to IAA. Code: N/A means that the official code is not available online; available code links are marked in red.

ages and mapping the visual features to annotated labels by training classifiers (aesthetically positive or negative) or regressors [Lu et al., 2014]. Although these methods have achieved great success, recent evidence reveals that the direct prediction of aesthetic scores removes the diversity (distribution) of human opinions [Talebi and Milanfar, 2018]. Some researchers have noted this limitation and proposed using the Earth mover's distance (EMD) loss [Talebi and Milanfar, 2018] to train the score distribution task, while others have attempted to adapt a personalized aesthetics model for individual preferences [Ren et al., 2017; Lv et al., 2021].

However, previous works ignore the importance of theme information, when humans assess aesthetics, they explicitly or implicitly take into account the influence of the themes; additionally, relying only on the weak supervision information of IAA datasets is not sufficient to understand aesthetics (Fig. 2). In this work, we focus on finding theme-adaptive understanding methods and investigating how to best use them for IAA to yield a competitive model.

3 Proposed Dataset

The emergence of new datasets has led to rapid progress on various IAA tasks. In this context, our goals for developing TAD66K are to provide a new challenge and to spark novel ideas to rethink IAA tasks. We describe details as follows.

3.1 Image Collection

In terms of data collection, we took the following steps. First, we collected themes that are widely popular among humans. To ensure the richness of these themes, we determined the most uploaded themes on the Flickr website from 2008 to 2021. With the help of the t-distributed stochastic neighbor embedding (t-SNE) technique [Van der Maaten and Hinton, 2008], we manually grouped the themes into seven superthemes, namely, plants, animals, artifacts, colors, humans, landscapes and other, which were further divided into 47 subthemes, as shown in Fig. 4.

Second, we reduced the long-tailed distributions of the annotated data by performing a preassessment during the data collection phase. Imbalanced data were ignored in previous works; for example, the AVA dataset, one of the largest aesthetic datasets currently available, contains annotated image aesthetic scores from 1 to 10, but we find that the majority score range (5∼6) has 2700 times more samples than the minority score range (9∼10), models trained on imbalanced data are biased toward majority examples. To alleviate the disadvantages of previous datasets, we preliminarily assessed the aesthetics of samples as good, general, or bad and tried to ensure that the samples are evenly divided among each of these three labels to maximize the diversity of the overall score.

3.2 Image Annotation

We set the annotation score of each image to range from 1 to 10, representing the range from the lowest aesthetics to the highest aesthetics. According to some theories of photography, different themes follow different evaluation criteria (see our supplementary material), so we labeled each theme separately. A common problem of subjective rating experiments is that the annotators often have no baseline (or reference) for assessing the measured effect. This often causes the subjective responses obtained before the annotators have seen adequate examples to be unreliable. To alleviate this, we provided anchor images for different rating scales as references and employed batch annotation (batch = 50). During the annotation process, the annotators needed to browse the anchor images and 50 images completely before evaluating the images. Finally, through manual cleaning, approximately 1200 opinions were collected on average for each image. We calculated the average value as the ground truth of the image.

4 ProposedModel

Overview. The general IAAmodel is described as follows:

where F indicates the IAA model, p represents the predicted aesthetic score, and x represents the input image. However, it is difficult to use only single aesthetic scores as supervision information to extract perceptually consistent features. Nevertheless, humans visually perceive and evaluate aesthetics after the image theme is recognized. Furthermore, the weight parameters θ ignore how theme variations influence the method with which aesthetics are perceived. With the above problems in mind, the proposed TANet model has three essential components (Fig. 5), which we describe as follows.

4.1 Theme Understanding Network

To extract theme features via direct supervision, we use ResNet18 as the backbone S and train it on the scene database [Zhou et al., 2017], achieving a 85.03% top-5 accuracy. The scene database is a repository of 10 million images annotated with 400+ unique theme semantic categories and environments, and almost the themes in TAD66K dataset and daily life are covered. It is worth noting that to prevent a gradual decline in the perception ability (Fig. 3) during IAA model training, we freeze the parameters at all times. We define the output of S as S(x), which is divided into two flows for processing. One flow is sent to a parameter generator L1 to adaptively generate θtheme (weights and bias):

where δ represents the parameters of L1 . To deal with the noise in S(x), the other flow is sent to the feature preprocessor L2 to reduce the spatial redundancy in the latent representation. Finally, we use a linear layer L3 to multiply the two flows to obtain the final output, and the whole process can be expressed as:

Thus, xtheme includes both the basic theme information and the rules governing how to perceive this information.

4.2 RGB-distribution-aware Attention Network

The second component extracts high-level color features from the RGB space to improve the understanding of the theme. The color distribution [O'Donovan et al., 2011] constitutes important information in aesthetics and has a close relationship with the theme. In terms of capturing this distribution, the standard self-attention mechanism [Vaswani et al., 2017] can coordinate the relationships between a token and all other tokens in natural language processing (NLP) tasks. However, for IAA tasks, the original information of the color distribution and similarity exists in low-level features closer to the original image, and a large feature map size means that there are more tokens; thus, global computations are of quadratic complexity [Liu et al., 2021].

In our implementation, we apply two improvements to the standard self-attention mechanism. We first split the input into nonoverlapping patches, where each patch is represented by a center point and is set as the average of the raw pixel RGB values. For images of any size, the patch space is composed of k × k central points, which results in a linear and low computational complexity to the input size. Specifically, we set k = 12. Second, we extract only the relationship of patches without multiplying them by the input. Let fab, fuv represent two centers; then, the output xrgb of our attention can be described as:

where ∥Nl=1 is the concatenation of RGB channels and Ql , Kl and d are the query, key and dimension, respectively, generated from the input in the standard self-attention. After extracting this relationship, we do not multiply it by the value (V ) but instead send it to the softmax layer after processor L4 to reduce redundant information and obtain the final result xrgb. As a result, the proposed attention mechanism endows the RGB space with a perceptually meaningful measure of the color similarity and distribution, as shown in Fig. 6.

4.3 Aesthetics Perceiving Network

The third component performs two functions. First, APNet directly extracts aesthetic features xaes from the input x; we use MobileNetV2 as the backbone, and the output is processed by L5 . Second, three features are fused to predict an aesthetic score, and the output is processed by L6 . We describe the whole process as:

where θaes represents all the parameters of Faes. Please refer to the supplementary material for more details.

5 Experiments

5.1 Experimental Settings

Evaluation Metrics. To evaluate the performance, we adopt two popular evaluation metrics, namely, Spearman's rank correlation coefficient (SRCC) S [Talebi and Milanfar, 2018] and the linear correlation coefficient (LCC) L [Talebi and Milanfar, 2018]; the evaluation indicators for the AVA loss also include the EMD loss E [Talebi and Milanfar, 2018], while adopting the SRCC/accuracy ratio R [Hosu et al., 2019] instead of the binary classification accuracy to better evaluate the generalized performance. For TAD66K, we adopt the mean squared error (MSE)Mas the loss function.

Benchmark Models. We select 17 SOTA models on the AVA dataset that 1) are recently published and 2) have a representative pipeline and apply 10 models (whose code is available) to our TAD66K dataset. There are 4 other designs specifically for personalized IAA that we use to evaluate performance on the FLICKR-AES dataset.

S ↑ L ↑ M↓ Speed ↑

0.440 0.457 0.020 13.89 it/s

0.483 0.513 0.017 10.03 it/s

0.462 0.478 0.019 13.71 it/s

- out memory

(AP + RGB + TU)Net 0.513 0.531 0.016 10.01 it/s

5.2 Performance Comparison

Performance on TAD66K. Compared with the 10 SOTA methods (Table 3), our TANet model achieves the best performance on all metrics. This suggests that understanding

the theme of an image fundamentally assists in perceiving the aesthetics of the image, especially when there are a wide variety of themes that may have different evaluation criteria. Some examples are shown in Fig. 1. In addition, a real-time inference video is available in the supplementary material.

Performance on AVA. Based on the overall performance of the 17 representative methods reported in Table 3, TANet still achieves the top performance on the popular AVA dataset. This is because its specially designed TUNet and RGBNet can automatically learn general high-level theme features and color features, which are crucial for better capturing the general evaluation criteria for a general aesthetic assessment.

Performance on FLICKR-AES. Previous work has shown that people with different personalities may prefer images with specific themes [Li et al., 2020; Ren et al., 2017]. When our network understands the theme information, it is able to notably improve the overall personalized IAA performance. Table 4 shows that our model achieves the best SRCC of 0.717, surpassing the second-best results by +7.2% SRCC, which means that our model can use a small amount of data to learn more about individual users' aesthetic preferences.

5.3 Ablation Study

We first examine the effectiveness of TUNet (Table 5), which obtains an SRCC of +9.8% and an LCC of +12.3%, while the MSE loss decreases by 15.0%. However, TUNet needs to extract theme features, which reduces the training speed of the model to 3.68 it/s. Furthermore, when RGBNet is added, the performance achieves increases in the SRCC and LCC of +5.0% and +4.6%, respectively, and the MSE loss value is decreased by 5.0%. Moreover, the proposed attention mechanism in RGBNet model does not significantly reduce the speed of the network. We also conduct a comparison test with the standard self-attention and find that due to its excessive computational complexity, it cannot work properly in the same way, which shows that proposed architecture is more computationally efficient than the traditional architecture. Finally, we add all elements. The SRCC and PLCC values are further improved to their highest values, which are +16.6% (SRCC) and +16.2% (LCC), while the MSE loss value is decreased by 20.0%, which confirms that the proposed elements are effective in promoting the model's performance.

5.4 Model Interpretation

Adaptive Weights. To discern whether the model adaptively adjusts the weights of theme feature fusion, we extract the weights of θtheme in Eq.(2), which includes 100 hidden nodes. We verify the trained TANet model on the test set of TAD66K, count the index of each image when it is activated in the maximum positive or minimum negative direction, and plot the results in Fig. 7. Interestingly, images with specific themes are activated by specific hidden nodes. e.g., TANet tends to assign lower negative weights (close to -1) to images with low aesthetics since these images usually have chaotic themes or are too dark. Among the most frequently visited nodes, images with harmonious colors and bright themes have more positive responses. Additionally, we find that some hidden nodes respond only to specific themes, such as images with the theme "art". This verifies that our model has learned to build the corresponding evaluation criteria.

Class Activation Maps. To visualize saliency maps, we fuse the 2-D feature maps of the last layers of the three elements in TANet, and we do the same for the other 10 SOTA methods with multiple branches. The attention of our model (Fig. 2) is able to cover more highly theme-correlated areas and is more in line with human perception. Furthermore, our model effectively solves the problem of attention dispersal.

6 Conclusions

This paper addresses the long-ignored influence of theme variation in IAA. To achieve this goal, we create a themeoriented TAD66K dataset (containing 47 themes), build up a complete benchmark (including the top 17 SOTA models) and develop a baseline model called TANet. Compared with the existing datasets, TAD66K is more challenging and more densely annotated; moreover, the proposed TANet introduces adaptive perception methods to extract theme features and achieves SOTA performance on three representative datasets. We hope our contributions will motivate the community to rethink IAA and stimulate research with a broader perspective.