Phonovisual Biases in Language: is the Lexicon Tied to the Visual World?

1 Introduction

How do words refer? How is a sound of the vocal tract mapped onto an object, event or state that is external to the mind? The conventional perspective on vocabulary structure, rooted in the structuralist tradition [Saussure, 1964], advocates an arbitrary origin of the lexicon, where words would have developed historically as a cultural and social product and passed along by tradition [Bloomfield, 1994]. However, the denial of any correspondence between linguistic sounds and their denotation is inherently problematic: seeing the lexicon as an arbitrary cultural product prevents any attempt of inquiry into its structure and its relationships with other aspects of human biology, irrevocably confining it outside of the scope of scientific explanation [Allott, 2001].

An alternative to arbitrariness is iconicity, the idea that phonemes can convey meaning per se, i.e. not only through contrastive relations with other sounds but also through their intrinsic sound qualities. While several modern linguistic theories reject iconic principles at the lexical level, they endorse them at the syntactic level with various degrees of explicitness, acknowledging a parallelism between the structure of language and various facets of the structure of experience [Levinson, 2000]. For instance, sequences of forms tend to correspond to sequences of occurrences [Bybee, 1985; Perniss et al., 2010]: in the sentence "I studied, walked, and showered" the reader will typically assume that the three actions were performed in the order in which they were uttered. Widening our focus to non-arbitrariness in general, it is safe to state that most linguistic theoretical frameworks do not posit that languages can be structured without any accountable principle (see for instance Bickerton [2012], Chomsky [2002]). Nonetheless, the idea of a principled systematicity of any kind in the lexical domain is sometimes dismissed as a doubtful exception at best. As noted by Allott [2001], the tension between the syntactic schematicity and the alleged lexical randomness raises the question of how a functional interface between these domains is even possible.

∗Contact Author

The idea that phonemes can carry inherent meaning had a long-standing tradition in philosophy before being nearly eradicated by the Saussurean axiom of the arbitrariness of the sign (see Magnus [2013] for a historical introduction). This fascinating hypothesis was revived in the late 1920s by two remarkable studies, which evidenced a surprising link between the participants' intuitions about a figure's name and its size [Sapir, 1929] or its shape [K¨ohler, 1929]. In the first study, participants were asked to match two tables of different sizes with the non-words 'mil' and 'mal', showing that the phonetic sequence containing the phone [a] was four times more likely to be associated with the larger object than the one containing the phone [i]. In the second study, when asked to match two novel shapes with the non-words 'maluma' and 'takete', English-speaking adults tended to label as 'maluma' the curled shape, and as 'takete' the sharp one. The latter experiment on shape symbolism had a higher resonance than the former in the cognitive science community: several studies exploited slight variations in the shapes and the phonetic forms of the non-words, and repeatedly demonstrated the psychological reality of the so-called 'maluma-takete' effect [K¨ohler, 1947; Werner, 1948], or 'bouba-kiki' effect, referring to the stimuli used by Ramachandran and Hubbard [2001]. These research efforts paved the way to a number of inquiries that replicated the former phonovisual correspondences in various geocultural contexts [Bremner et al., 2013; Ramachandran and Hubbard, 2001] and at different developmental stages [Maurer et al., 2006; Ozturk et al., 2013].

The results of the studies that employed explicit matching tasks were complemented by a body of findings that succeeded in identifying an implicit cross-modal correspondence between phonological form and shape [Hung et al., 2017; Sidhu and Pexman, 2017], suggesting that the aforementioned effects might be pre-semantic and presumably lexical, at least in part. Besides shape and magnitude symbolism, other visual attributes that showed a correspondence with the respective phonetic linguistic sign are colour [Johanssohn et al., 2020] and lightness [Hirata et al., 2011].

Iconicity is not limited to oral languages: a growing weight of evidence is disclosing analogies between the form of a considerable subset of signs and their referents [Emmorey, 2001; Kuhn, 2020; Schlenker and Lamberton, 2012]. Neither it is a privileged attribute of human languages, as it has been documented also in bee dances [Berm´udez, 2007] and primate calls [Burling et al., 1993]. Recently, linguistic iconicity has gone from being a marginal - although appealing - matter to being integrated into broader theories of language evolution [Ramachandran and Hubbard, 2001], processing [Lockwood and Tuomainen, 2015] and acquisition [Asano et al., 2015; Imai et al., 2008]. Rejecting the assumption of a totally arbitrary mapping between referens and referent sensibly reduces the problem space of language emergence, establishing constraints on the consensus in word choice. Hence, the reviewed findings on phonosymbolic mappings could provide a pivotal clue for understanding the origins of proto-languages, suggesting that there may be natural constraints on the modalities in which objects are referred to through sounds. Furthermore, in the context of language learning, a propensity to induce an iconic referentiality of the linguistic sign alleviates both Quine's logically insurmountable problem of linking the phonological form of a novel word with its meaning [Quine, 1960], and the speech segmentation (or word discovery) problem, i.e. the initial difficulty in the localization of word boundaries in a continuous speech stream without the knowledge of any word. Iconic links would then help children learn perceptually grounded semantic concepts, and discover structures across spoken and contextual input. Phonovisual correspondences have been shown to affect different cognitive faculties, such as memory [Ramachandran and Hubbard, 2001], categorization [Lupyan and Casasanto, 2015], and emotion recognition [Slavova and others, 2019]; moreover, they exert an influence on actional processes such as phonatory behaviour [Parise and Pavani, 2011], spatial navigation [Rabaglia et al., 2016], and hand grip [Vainio et al., 2013].

Iconic sound-to-referent relations have been disclosed in different perceptual modalities, such as haptic touch [Fryer et al., 2014; Graven and Desebrock, 2018], kinesthesis [Fontana, 2013], and taste [Gallace et al., 2011]. Nonetheless, vision seems to hold a privileged relationship with the phonetic representation of the denotatum: cross-modal correspondences in the olfactory-gustative modality do not seem to exhibit cross-cultural consistency [Bremner et al., 2013], and phonotactile biases might be at least partially mediated by the role of visual imagery [Fryer et al., 2014]. Hence, the present work aims to explore the multifarious topic of lexical iconicity with the physical attributes of the referent intended as its visual features.

Within the computational framework, the analysis of iconic biases has mainly followed two general trends [Guti´errez et al., 2016]: a localist approach, aimed at identifying some islands of non-arbitrariness in language [Abramova et al., 2013; Sagi and Otis, 2008], and a global program, directed toward an assessment of the pervasiveness and systematicity of linguistic iconicity [Dautriche et al., 2017; Monaghan et al., 2014; Tamariz, 2008; Pimentel et al., 2019]. Our work fits into the second trend, and aims to extend the previous findings through an exploration of iconic regularities beyond the limits of a single language.

To our knowledge, few studies have tackled the topic of sound symbolism from a cross-linguistic perspective; those that have done so, have generally focused on a small set of concepts or words on a massively multilingual scale [Blasi et al., 2016; Wichmann et al., 2010] (although see Pimentel et al. [2019] and de Varda and Strapparava [2021] for lexiconwide studies). Our work, in contrast, aims to perform an analysis on phonosensory (and specifically phonovisual) correspondences on a representative sample of concrete objects in a selected set of languages. We evaluated the performance of a Long Short-Term Memory network (LSTM) in associating phonetic vector sequences with visual vectors picturing their referents, reporting an above-chance performance of the model on an unseen language. To the best of our knowledge, this is the first cross-modal and cross-linguistic study exploiting deep learning methodologies to find an iconic relation between a linguistic sound and the physical-geometrical attributes of its denotation. This effort provides an example of rewarding contamination between methodological advances driven by artificial intelligence research, and theoretical questions stemming from philosophy, linguistics and psychology.

2 Methods

In the present study, an LSTM-based Recurrent Neural Network is trained to associate the phonetic representation of a word to the visual representation of its referent. Visual representations consist in the output of a pre-trained Hierarchical Convolutional Neural Network (HCNN) in response to a forward pass of a given image, whereas the phonetic features corresponding to each image's label are expressed as sequences of phonetic vectors in 22 dimensions. The experimental pipeline is summarized in the flowchart in Figure 1.

2.1 Dataset

Our analyses were performed on the THINGS database [Hebart et al., 2019], a resource that comprises 26,107 highquality naturalistic images depicting a set of 1,854 diverse object concepts. The concepts were sampled systematically from concrete picturable and nameable nouns in the American English language, and the corresponding images (12 or more for each concept) were extracted through a largescale web image search and cropped to square size. Each item of the dataset was composed by an image and a corresponding label, that were preprocessed independently as described in the following subsections. In order to limit the effect of morphological noise in the labels, we excluded from the database all the compound words before any subsequent analysis; with this exclusion criterion, the resulting dataset consisted of 22,268 images, depicting a set of 1,549 concrete words.

2.2 Translation

With the intention of maximizing the linguistic distance between the training and the test set, each image label in the dataset was translated in five languages belonging to five language families (see Table 1) 1 . In order to define a pipeline that could guarantee a high-quality translation for a sufficiently large amount of items, we first searched for lexical matches through word2word [Choe et al., 2020], a collection of bilingual lexicons constructed from the publicly available OpenSubtitles2018 dataset [Lison et al., 2018]; then, for the items for which a translation was not available through the aforementioned package, we employed the ground-truth bilingual dictionaries based on fastText, released by Facebook Research 2 [Conneau et al., 2017]. We then included in the analyses only the words for which a translation was available in all the languages considered in the study - in other words, the set intersection of the translated items. The resulting dataset consisted of 16,820 images, depicting a set of 1,161 concrete words. The percentage of translations obtained with each translation tool for all the languages considered in the study is reported in Table 1, along with the percentage of missing items.

2.3 Phonetic Representations

For each word in the multilingual dataset, we obtained its phonemic transcription with Epitran, a Python library for transliterating orthographic text in the International Phonetic Alphabet (IPA) format. Then, we converted the IPA string into a sequence of feature vectors with PanPhon, a package that traduces IPA segments into subsegmental articulatory features [Mortensen et al., 2016]. We agree with Jakobson and Waugh [2011] when they assert that "most objections to the search for the inner significance of speech sounds arose because the latter were not dissected into their ultimate constituents" (p. 182). Hence, we chose not to directly hotencode the IPA strings in order to allow the network to exploit the underlying similarities that make different phones more or less related to each other. For instance, [p] and [b] are similar in that they only differ in the feature [+/- voiced], whereas [t] and [u] differ by 13 subsegmental features. These internal similarities would have been lost with a raw hot-encoding over the IPA vocabulary, along with the information-rich representation offered by a phonetic feature decomposition.

1 Following the Omniglot genealogical classification of languages at https://omniglot.com/writing/langfam.htm

2 Publicly available at https://github.com/facebook research/MUSE

Before being loaded into the LSTM model, all the sequences were padded, with a maximum length of 15.

2.4 Visual Representations

The visual representations included in this study consisted of the outputs of the fifth max-pooling layer of the HCNN VGG16, a deep convolutional network for large-scale image recognition [Simonyan and Zisserman, 2015], in response to a forward pass of each image in the dataset. HCNNs consist of serial blocks of layers containing simple patterns repeated across the visual input. Despite the relative simplicity of every single layer, deep convolutional networks can compute complex transformations of the input data. We fed each image x in our stimulus set through the network to extract the resulting feature maps j(x) of block5 pool. The weights of the VGG16 network had been configured according to its pretraining on ImageNet. The outputs of the target layer were flattened before being processed by the LSTM model. We employed the output of the VGG16 network as a proxy of a representational format proper of the human visual processing system. Indeed, HCNNs have been employed to model different encoding steps of the mapping of visual stimuli to neural responses as measured in the brain, without being explicitly optimized to fit neural data [Yamins and DiCarlo, 2016].

2.5 Neural Architecture

An LSTM-based Recurrent Neural Network was trained to map the chains of phonetic feature vectors in input into the visual vectors in output. In contrast with standard feedforward neural networks, LSTMs are endowed with feedback connections, that allow them to process not only single data points, but also data sequences. The model was built with Keras, a deep learning framework for Python [Chollet, 2015]; it included a masking layer, followed by a single LSTM layer with 500 units, a dropout of 0.2 and a recurrent dropout of 0.2. The LSTM layer was connected to a dense layer with the number of units (25088) matching the dimensionality of the target visual vector, and equipped with the rectification nonlinearity (ReLU). Cosine similarity was used as both objective function and metric, and the Adam optimization method was employed for training [Kingma and Ba, 2014]. The hyperparameters were set without tuning.

2.6 Experimental Conditions

In defining the experimental conditions, we followed two main principles to limit to the furthest extent the effect of the etymological relatedness between the items in the training and the test set: we established for none of the images and none of the concepts to appear in both sets, and we applied the same constraint to the languages in which the concepts were translated. Therefore, we randomly divided the concrete concepts into two subsets with a train-test split ratio of 0.2. Following this partition, the training set consisted of 929 concepts depicted by 13,397 images, whereas the test set was composed of 232 concepts represented by 3,423 images. We trained six different cross-lingual models with identical configurations following a non-random 6-fold cross-validation procedure. Each model was trained to associate the phonetic vector sequences corresponding to the label's translation in five languages to the respective visual vector in the training set; then, it was validated on the samples of the test set in the language that was excluded from the training. Thus, the train and test sets in all the experimental conditions were disjoint with respect to the concepts, the images representing them and the languages from which the phonetic vectorization was extracted. In order to define a baseline for the evaluation of each model's performance, we compared its results with the ones achieved by a parallel model, trained on a dataset where the correspondence between phonetic and visual vectors had been randomized; in practice, the mismatch between input and output vectors was obtained by randomly shuffling the order of the visual vectors in the training set. We will refer to this manipulation as the random condition. All models were trained on 66,985 samples (13,397 × 5 languages) and tested on 3,423 items.

3 Results

Table 2 reports the results of the cross-lingual models paired with their random counterparts. The first column of the table specifies the language of the fold on which the validation was performed, implying that the training had been carried out on all the languages but the one in the test set, as specified in the previous subsection. The following six columns of the table present the mean, the standard deviation and the 95% confidence intervals of the cosine similarity between the target visual vector and the cross-lingual (cl) or the random (r) model's prediction for every item in the test set. We evaluated the statistical significance of our results through a set of paired samples t-tests between the element-wise cosine similarity of the target visual vector with the vectors generated by the two alternative models for each experimental condition. The eighth and ninth columns of the table present the t statistic and the associated p-value for each of the contrasts evaluated by the test. As a measure of effect size, we report in the last column of the table the Cohen's d relative to the standardized difference between the mean metric of the two alternative models.

Across all the experimental conditions, the cross-lingual models outperformed the randomized baselines, with all the comparisons reaching high statistical significance. Although there are considerable differences in the results of the alternative models reported in Table 2, the multilingual settings always yielded higher performance scores with respect to the randomized baselines. The confidence intervals of the cl models do not overlap with the intervals of the r models, suggesting that the phonovisual correspondences in the lexicon can be learned in any direction and generalized to all the languages included in our analyses. Additionally, the measures of effect size suggest a medium-to-large difference across models, with the only marginal exception of the zeroshot transfer to Vietnamese.

4 Discussion

The LSTM network trained on multilingual data showed the ability to induce cross-linguistic regularities in sound-tovision mappings, suggesting that linguistic data alone contain the sufficient amount of information to encode for phonovisual biases. Hereby, we wish to clarify our intentions in using the two neural architectures for processing the images and their link to linguistic sounds. Our purposes were structurally different for the HCNN VGG16 and the LSTM: in the former case, we employed the convolutional neural network to transform the raw RGB images in input into cognitively inspired visual representations; hence, the neural network was intended as an approximation of the human perceptual system. In the latter case, we adopted the LSTM-based Recurrent Neural Network in order to uncover hidden correspondences between two different representational formats - a task that arguably requires the use of complex transformations.

While the behavioural studies presented in the Introduction disclosed a strong and consensual link between meaningless speech sounds and magnitude, colour and geometrical shape, we aimed to explore the link between meaningful speech sounds and visual representations. In the light of the evidence that highlights the role of visual imagery in shaping cross-modal phonosymbolic biases [Fryer et al., 2014], we built an imaginative network trained to elicit a visual representation relying on a word's sound. The network was shown to learn a generational process where the images produced in response to the phonetic inputs resembled their actual referents more than what would be expected by chance. Our strict manipulation of the linguistic distance between the languages in the training and in the test set allows us to rule out the effect of any etymological relatedness between the different languages' vocabularies.

We believe that the central interest of our findings resides in the learning process being cross-linguistic, yielding generalizations that are language-independent. Indeed, crosslingualism is a crucial testbed for the purpose of identifying a universal sound-symbolic substrate underlying all languages, as opposed to language-specific idiosyncratic systematicity; whether this symbolic underpinning is to be considered innate [Ramachandran and Hubbard, 2001] and possibly reminiscent of a primordial state of neonatal synesthesia [Maurer and Mondloch, 2005] or implicitly induced by statistical correlations in shared sensorimotor experiences [Ernst, 2007; Spence, 2011] is beyond the scope and the explanatory power of the present study.

5 Conclusion and Future Directions

The present paper aims to contribute to the growing body of evidence against a purely cultural and arbitrary origin of the lexicon. Its purpose is to demonstrate that the vocabulary is profoundly entangled with the visual world, and that the correspondence between linguistic sounds and visual representations is a candidate universal feature underlying wordformation, shared across languages and language families. Moreover, it shows that this correspondence can be efficiently captured by a computational system in a selected subset of the lexicon. Crucially, the regularities detected by the network were not language- or item-specific, since in each experimental condition the items in the test set consisted of novel concepts, expressed in a previously unseen language and depicted by original images.

We wish to highlight the fact that the present work does not disclaim in any way the significance of a certain degree of arbitrariness as a fundamental property of language: without dissociating form and meaning it would not be possible to denote a potentially infinite set of concepts [Lockwood and Dingemanse, 2015]. Iconic and arbitrary principles are not mutually exclusive [Sidhu and Pexman, 2018]: on the contrary, they complement each other enriching the lexicon with their specific qualities.

Once the pervasiveness of lexical iconicity in the concrete vocabulary is acknowledged, a number of questions must be addressed to properly understand the phenomenon under scrutiny. Substantiating the claim of a cross-linguistic coupling between phonetics and vision was an imperative step, but it does not inform us on the locus of the biases that shape the cross-modal correspondences. We leave for future research an assessment of the precise visual attributes that show a privileged link with certain phonetic features, as well as an analysis of the relative contribution of shape, magnitude and colour to the phonovisual correspondences.

Acknowledgements

We thank Chiara Saponaro, Raffaella Bernardi and Stella Frank for their valuable insights, suggestions and comments.