Towards Contextually Sensitive Analysis of Memes: Meme Genealogy and Knowledge Base

1 Introduction

Memes are a unique social phenomenon which have grown in popularity as social media use increases. They represent a unique type of multimodal online content constructed from recycled images or catchphrases, referencing cultural icons or ideas, and invoking strategies such as humour and sarcasm to communicate simple or complex ideas. Many memes use image and text formats where the combination of each mode provides context for the other, and they are constructed by users continuously referencing and exploiting other meme content. Their reliance on contextual knowledge and referentiality makes them difficult to analyse using computational methods which lack human context.

2 Related Work

Current areas of interest in meme analysis are classification, AI meme generation and sentiment analysis. Meme classification refers to tasks which seek to distinguish memes from non-meme content or identify binary classes in memes - the most prominent recent example being the Facebook Hateful Meme Competition [Kiela et al., 2021]. The competition dataset required multimodal approaches as the interplay of text and image could change the final classification; the winning solutions proposed extracting features from visual streams and tagging these with external labels. Other studies proposed similar multimodal approaches with object detection, feature identification and classification of meme popularity [Beskow et al., 2020; Barnes et al., 2021]. However, Kirk et al. [2021] noted that models from the Hateful Meme competition performed poorly on 'wild' meme classification as models were trained on an artificial dataset.

Developments in AI meme generation proposed multimodal solutions and text-image alignment to match generated text to the correct template but demonstrated limited understanding of textual or visual codes in memes by confusing common linguistics structures of one meme template with another [Sadasivam et al., 2020; Peirson and Tolunary, 2018]. Wang and Wen [2015] proposed a non-paranormal approach to generating meme descriptions with an external knowledge module from reverse image search, demonstrating AI-generated memes that used puns.

Meme sentiment tasks are approached through natural language processing and aim to achieve fine-grained emotion classification. Multimodal approaches were again developed as part of the SemEval 2020 Task 8 analysis of the Memotion dataset, a prelabelled dataset with fine-grained emotions, outperforming unimodal approaches. However, performance was still overall poor on sentiment analysis task for all models [Sharma et al., 2020].

3 Research Objectives

This research will contribute computational approaches and techniques for meme analysis which are contextually sensitive and able to extract latent meaning with three objectives.

Classification of memes and extraction of features will be achieved as part of the proposed meme genealogy which differs from current research by identifying prominent features in memes through computer vision and natural language processing, to contribute a comprehensive mapping of meme relationships via their identified features.

The construction of a meme-specific ontology and knowledge base will collate contextual information from online sources, drawing on the most successful models in prior research which exploit external knowledge to augment models. The proposed knowledge base will use features inherent in memes and match them with contextual knowledge from online sources to ensure the connotations and references that have been constructed as memes are continually produced is captured.

The final object will use this knowledge base to tag memes with relevant semantic and contextual knowledge. Exploiting 'meme families' and entity relationships developed in the genealogy would allow for minimal annotation of only a few meme templates or features for future analysis, and can additionally be applied to analysing new, emerging memes based on their shared features to known memes.

4 Proposed Methodology

The proposed methodology draws on the achievements identified in current meme research: multimodal approaches; external knowledge; and fine-grained sentiment/discourse analysis applied to both the image and text as opposed to only text in previous studies. The latter will be solved using a semiotic framework to analyse visual/textual codes in memes but is expected to be achieved after the construction the knowledge base. The first stage explored is the construction of a meme genealogy and taxonomy, which will inform the development of a meme-specific knowledge base and ontology. Figure 1 illustrates the first stage and the incorporation of external knowledge:

5 Conclusion and Contributions

This research briefly outlines a proposed method towards contextually sensitive analysis of memes through a structured meme genealogy and knowledge base. It contributes a potential approach to accurate classification and sentiment analysis of memes by incorporating the rich history of referential meaning generated by users continuously creating memes, alongside methods to annotate memes with feature-inheritance. Currently, the first pass has been achieved and work is ongoing to extract features for the knowledge base.