AI-Assisted Human-Pet Artistic Musical Co-Creation for Wellness Therapy

1 Introduction

The increasing prevalence of pet ownership has transformed the perception of animals from mere companions to integral members of the family unit. This shift has sparked interest in exploring innovative therapeutic modalities that leverage the unique bond between humans and their pets.

Recent advances in animal-assisted therapy (AAT) have demonstrated significant psychosocial benefits, particularly in reducing anxiety and improving social engagement in ASD (Autism Spectrum Disorder) populations [Sissons et al., 2022]. However, traditional AAT modalities predominantly focus on tactile interactions (e.g., petting), while auditory cocreation remains underexplored.

Current music therapy paradigms primarily focus on therapists playing music based on user feedback [Hunter et al., 2011], AI algorithms generating music according to user emotions [Wang et al., 2024c], and involving users in the music creation process through gamification [Hahn et al., 2006]. However, the role of pets has not yet been incorporated into music therapy.

This paper introduces a pioneering approach to animalassisted therapy by proposing a novel framework for musical co-creation between humans and their pets. By integrating the vocal characteristics of pets into the music creation process, we aim to enhance emotional engagement and therapeutic outcomes, thus providing a fresh perspective on AAT. To achieve this, we have developed a system, PetCoCre, that incorporates three distinct functionalities: PetRhythm, PetMelody, and PetVocalia.

Beyond these three character paradigms, our technical innovation lies in proposing SaMoye, the first open-source, high-quality zero-shot SVC model. Existing open-source SVC methods struggle with zero-shot conversion due to incomplete feature disentanglement or reliance on speaker look-up tables. SaMoye disentangles singing voice features into content, timbre, and pitch. Furthermore, we enhance timbre features by unfreezing the speaker encoder, and mixing the speaker embedding with those of the top-3 most similar speakers. We also collected and established a large-scale singing voice dataset (comprising 1,815 hours of pure singing voice and 6,367 speakers) to ensure zero-shot SVC performance. We conduct objective and subjective experiments to find that SaMoye outperforms other models in zero-shot SVC tasks under extreme conditions like converting singing to animals' timbre.

Furthermore, we conducted a series of experiments to validate the efficacy of this co-creation paradigm. By engaging participants in collaborative music-making with pets, we aimed to assess the impact of this interactive experience on emotional well-being, particularly in alleviating anxiety and enhancing pleasure. The results show that participants' creative satisfaction and pleasure were significantly enhanced, and their anxiety levels were markedly reduced, underscoring the potential of this interdisciplinary approach to address pressing social issues.

Open-source links: Code 1 and Model Checkpoint 2 .

2 Background

2.1 Current Status and Potential Value of Animal-Assisted Therapy

Pandey et al. conducted a systematic study of the qualitative and quantitative evidence of AAT's role in enhancing patients' well-being, finding that it significantly reduces anxiety and improves overall quality of life [Pandey et al., 2024]. Flynn et al. reviewed how AAT can improve engagement in behavioral and mental health services, highlighting its ability to attract patients and increase treatment adherence [Flynn et al., 2022]. Sissons et al. provided a systematic review of AAT for improving social functioning in children with autism, noting that interactions with animals, particularly horses, can enhance social skills and reduce anxiety [Sissons et al., 2022]. Dimolareva and Dunn (2021) conducted a meta-analysis showing that AAT can improve social skills and communication in children with ASD (Autism Spectrum Disorder), though the effectiveness varies by intervention type [Dimolareva and Dunn, 2021].

Traditional AAT modalities predominantly focus on tactile interactions (e.g., petting), while auditory co-creation remains underexplored despite music therapy's proven efficacy in neurorehabilitation. De Witte et al. demonstrated through a systematic review and meta-analysis that music therapy is highly effective in reducing stress and improving mental health [De Witte et al., 2022]. Barnett and Vasiu reviewed the neural mechanisms behind the therapeutic effects of creative arts, showing how they can enhance psychological and physical health [Barnett and Vasiu, 2024].

1 https://github.com/CarlWangChina/SaMoye-SVC

2 https://huggingface.co/karl-wang/SaMoyeSVC/tree/main

2.2 Existing SVCModels and Their Limitations

SVC (Singing Voice Conversion) aims to convert the timbre in a given song to the reference audio without disrupting the original content. This technique has wide applications such as virtual singers, music production and other artistic domains which currently experience considerable growth thanks to the advances in AI such as AI-based music and art [Wang et al., 2022; Wang et al., 2024b; Wang et al., 2024a; Mao et al., 2023].

SVC methods predominantly rely on the recognitionsynthesis framework, which involves recognizing the content of the singing voice and then synthesizing it in the target voice. [Nercessian, 2021] use a pretrained LSTM to extract the speaker embedding and concatenate it with the original speaker's phoneme and loudness embedding in the decoder to generate the converted audio. PitchNet [Deng et al., 2020] introduces a singer prediction network and pitch regression network to control the timbre and pitch stability. The embeddings from the two networks are fed into the decoder with the output from the encoder to generate the audio. [Luo et al., 2020] use separate encoders to extract singer and techniques embedding for singer and techniques classification tasks respectively and are concatenated before feeding into the decoder and refinement network to generate the converted audio. [Polyak et al., 2020] takes as input the speech features by a pre-trained automatic speech recognition (ASR) model Wav2Letter, the F0 feature by Crepe, and the loudness feature from the power spectrum. The speaker embedding from the target singer is included in the generator for converted audio generation. [Li et al., 2021] introduce F0 features, PPGs features as content features, and the Mel-spectrogram as the timbre features, which is enhanced through singer classification and reconstruction. FastSVC [Liu et al., 2021b] leverages sine-excitation signals and loudness features and uses a Conformer model to extract content features. These studies introduce HiFi-GAN and BigVGAN as the vocoder to generate the converted audio. VITS refers to Variational Inference with Adversarial Learning for End-to-End Text-to-Speech.

3 Method

3.1 PetCoCre SystemOverview

This paper aims to explore the cognitive processes underlying the enhancement of user emotional engagement through the use of "pet timbre" as a digital avatar, utilizing theories of social presence and examining the psychological mechanisms involved in anthropomorphic projection. We introduce PetCoCre, a human-pet musical art co-creation system, which defines three paradigms for incorporating pet sounds into AI-driven music creation: (1) PetRhythm: Rhythmic percussion through beat synchronization, where pet sounds are used as percussive instruments; (2) PetMelody: Melodic instruments via pitch-shifting alignment, employing pet sounds as primary melodic instruments; and (3) PetVocalia: Vocal synthesis combining pet timbre with human semantic content, which involves extracting pet timbres and performing sing voice conversion.

3.2 PetRhythm

PetRhythm involves the use of pet sounds as rhythmic percussion instruments, synchronized with beats to create a unique musical texture, thereby enhancing the rhythmic complexity and richness of the music. We employ an real-time beat recognition algorithm to align the pet vocalizations with the musical beats, effectively using them as rhythmic percussion instruments. To enhance the richness and naturalness of this alignment, we introduce a stochastic model that simulates the natural fluctuations in biological rhythms by incorporating randomness in the intervals between beats and vocalizations.

3.3 PetMelody

PetMelody transforms pet sounds into melodic instruments by aligning their pitches, allowing pets' voices to serve as primary musical elements, thus integrating pet vocalizations seamlessly into the melodic structure of compositions.

We first utilize the SheetSage algorithm to extract the main melody in the form of a MIDI file from a chosen audio song. Subsequently, the median F0 of each pet sound is transposed to match the corresponding MIDI note pitches found in the extracted melody. This requires converting the pet's natural vocal frequency into the nearest equivalent MIDI note, thereby assigning it a specific pitch within the scale. For example, if a dog's bark corresponds to a fundamental frequency of 440 Hz, this would be matched to MIDI note A4, which also has a frequency of 440 Hz. Finally, we map out the duration and beat placement for every pet sound based on its corresponding MIDI note and integrate them seamlessly into the melody, thereby making the pet sounds an integral part of the composition.

3.4 PetVocalia

In PetVocalia, the challenge of converting pet vocal timbres, which are generally 'unseen' data by conventional Singing Voice Conversion (SVC) models, necessitates a zero-shot SVC approach capable of timbre conversion from short reference audio without fine-tuning. However, existing opensource SVC models typically perform poorly on such zeroshot tasks, often requiring minutes to hours of singing data for fine-tuning. Given the absence of open-source zero-shot SVC models that met our performance requirements, we developed SaMoye model by modifying, optimizing, and scaling up the training of the Whisper-VITS-SVC 3 architecture.

Overall Architecture

The overall architecture is illustrated in Figure 2. Considering the importance of pitch in singing, we use RMVPE to extract pitch features. We use GE2E as the speaker encoder to extract speaker embedding from the audio as the timbre features. We use a Flow model to obtain the prior distribution from these features, while another posterior encoder derives the posterior distribution from the original waveform and speaker embedding. SaMoye is trained on audio reconstruction by aligning the posterior distribution with the prior distribution through minimizing their KL divergence. We also introduce a multiscale discriminator for adversarial learning. In the inference stage, we extract content features and pitch features from the original audio, and timbre features from the reference audio. These are used to obtain their prior distribution via the Flow model, which then generates the converted results. For timbre features, instead of freezing the speaker encoder during training, we train it jointly to enhance the timbre information in the speaker embedding. For content features, we fuse the features from existing ASR models, HubertSoft and Whisper.

For content features, we fuse the features from existing ASR models, HubertSoft and Whisper. We use a Flow model to obtain the prior distribution from these features, while another posterior encoder derives the posterior distribution from the original waveform and speaker embedding. SaMoye is trained on audio reconstruction by aligning the posterior distribution with the prior distribution through minimizing their KL divergence. We also introduce a multi-scale discriminator for adversarial learning. In the inference stage, we extract content features and pitch features from the original audio, and timbre features from the reference audio. These are used to obtain their prior distribution via the Flow model, which then generates the converted results. Instead of freezing the speaker encoder during training, we train it jointly to enhance the timbre information in the speaker embedding.

3 https://github.com/PlayVoice/whisper-vits-svc

Timbre Leakage Problem

Some early studies introduced speaker look-up tables for timbre features, making it difficult to extend the table for unseen speakers. More recent studies dropped speaker look-up tables and trained a speaker encoder to extract speaker embeddings from a given audio. However, this led to the problem of timbre leakage in content features. Content features are extracted by pre-trained automatic speech recognition (ASR) models such as Hubert, Whisper or ContentVec. Although these ASR models preserve semantic information while reducing timbre information in content features, timbre leakage when faced with unseen data can result in the converted audio being more similar to the original than the reference audio.

Adversarial Learning

We introduce an adversarial training strategy to train the generator to synthesize converted audio and a discriminator to distinguish between generated and real audio. For the generator, the Mel-spectrogram and speaker embedding pass through the posterior encoder to get the posterior latent variables zq. We apply the BigVGan decoder to generate the audio from the latent variables. Meanwhile, the pitch and content features pass another encoder to get the latent variables zp. We use the speaker embedding as the condition for a Flow model to generate the prior latent variables zt to be aligned with the posterior latent space. We compute the Kullback-Leibler divergence (KL) between the mean µ and variance σ2 of zt and zq, as well as zf and zp, and sum them up as Lkl. We also sum the L1 and L2 loss for the waveform as Lwav and for the mel-spectrogram as Lmel for audio reconstruction. We also use Lstft following [Takaki et al., 2019].

For the discriminator, we use the multi-scale and multiperiod discriminator in our study, which takes the generated and real audio as input to compute the L1 loss between their feature maps in the discriminator as Lfmap. The loss for the discriminator is shown in (1) and the adversarial loss for the generator is described in (2), where D is the discriminator, and a and a′ are real and generated audio, respectively.

The final loss for the generator is shown below, where we set α to 1.0, β to 0.2, and γ to 9.0 during training.

Retrieval Inference Strategy

We are inspired by retrieval-based voice conversion to introduce another inference strategy shown in Figure 3. In the inference stage, SaMoye retrieves the top-3 most similar speaker embeddings from a speaker embedding database and averages them with the reference speaker embedding. We build the speaker embedding database from the dataset to compute the cosine similarity between the reference audio and the embeddings in the database. Then, in the inference stage, we can retrieve the top three similar speaker embeddings and average them with the reference speaker embedding to serve as the timbre features. This method may reduce the timbre similarity of the converted audio but can improve zero-shot performance by exploiting seen speaker embeddings to fill the gap introduced by unseen reference audio, which can otherwise result in adverse outcomes such as mute voice.

Expanding Training Data

The primary limitation for zero-shot SVC task performance is the training data. SVC training data can be classified as parallel (multiple singers performing the same song) or nonparallel. Due to the limited availability of parallel data, existing models are typically trained on non-parallel data. The number of speakers in the base model significantly impacts zero-shot SVC performance, as zero-shot SVC requires models to see as much data as possible to enhance their generalization capability. Therefore, in addition to pet sounds, we have collected a substantial amount of human voice data to support subsequent training.

By collecting online audio and open-source datasets, we established a large-scale non-parallel SVC open-source singing and speech dataset comprising 1,815 hours and 6,367 speakers. We manually checked this dataset to ensure its quality. Specifics of these datasets are detailed in Table 1.

For data processing, all audio sampling rates are unified to 32kHz. When converting waveforms to Mel-spectrograms with 80 filters, we use a filter length of 1024, a hop length of 320, and a Hanning window of length 1024. This dataset includes 36.5 hours of online music, from which we separate pure human vocals using Demucs [D´efossez et al., 2019]. Subsequently, the obtained data are manually checked to remove any errors in recognition results.

Due to the inclusion of multiple singing datasets, there are balancing considerations regarding aspects such as language and style (e.g., differences in timbre and tone across various languages). As this is not the primary research focus of this paper, interested readers can refer to existing studies that have conducted relevant comparisons, such as [Creel et al., 2023].

4 Experiment Setup

4.1 Singing Voice Conversion Experiment Metrics and Participants

Since pet timbre conversion is inherently a zero-shot SVC task, and this particular task represents an extreme testing scenario that may result in performance degradation, we need to determine which model is best suited to serve as the base model for the PetVocalia functionality. We use several metrics to evaluate the quality of the converted audio and how similar the timbre is to the target pet timbre:

• Timbre Similarity: This metric is based on a percentage score, where 1 indicates identical timbre and 0 indicates completely different timbre. In this study, the metric is evaluated against the target pet timbre.
• Mean Opinion Score on Quality(MOS): MOS is a widely-used audio or video quality evaluation standard based on expert evaluation. The score of MOS is from 1 to 5, where a higher score means higher quality.
• Perceptual Evaluation of Speech Quality (PESQ): PESQ computes multiple perspectives like temporal alignment and perceptual filtering. PESQ score is from -0.5 to 4.5, where a higher score stands for better perceptual quality.
• Short-Time Objective Intelligibility (STOI): STOI represents how well the audio can be comprehended, and the STOI score is from 0 to 1. The higher STOI score means that the audio is easier to understand.
• Non-Intrusive Speech Quality Assessment (NISQA) 4 : NISQA is a non-intrusive metrics based on a pre-trained deep learning model. Given audio, the NISQA predicts MOS for audio quality, NOI for noise degree, DIS for audio coherence, COL for timbre quality, and Loud for loudness. All these predicted scores are the higher the better.

4 https://github.com/gabrielmittag/NISQA

We selected five song clips, each 20 seconds in length, and 11 speakers (6 humans and 5 pets) to evaluate the model's performance. These five songs cover the full range from bass to treble. The human timbres included 4 male and 2 female voices, while the 5 pet timbres consisted of 3 cat and 2 dog sounds.

For the subjective evaluation, we recruited 19 professional musicians. These participants (9 female, 10 male; aged 2137) are distinct from those involved in the subsequent humanpet co-creation experiment. They listened to the original music and target timbres, and then evaluated the converted song clips. We have documented their areas of musical specialization (e.g., electronic music production, musicology, music education). An anonymized list of these participants, with names redacted for privacy reasons, can be provided in the appendix.

4.2 Human-Pet Co-Creation Experiment Metrics and Participants

Our experiment adopts a dual-group control design involving 30 healthy adult participants. This design enables comparative analysis of response differences between groups under various conditions. Each participant was instructed to engage in AI music creation under different paradigms. These sessions could not be conducted consecutively on the same day, in order to avoid carryover effects between the three paradigms that might result from continuous sessions. Before and after each creation session, participants were required to complete web-based questionnaires to record relevant metrics. Three key psychological scales were selected to assess subjective experiences:

• Creative Pleasure: Measured using the PANAS-X (Positive and Negative Affect Schedule - Expanded Form), particularly its Positive Affect subscale, containing 10 semantic differential items (e.g., excited-calm, inspired-dull).
• Anxiety Reduction: Evaluated through the state subscale S-AI of STAI (State-Trait Anxiety Inventory), adapted for music creation contexts with 12 retained items. It assesses anxiety in various populations, including medical and psychiatric patients, occupational groups, and evaluates psychotherapy and medication effectiveness.
• Pet Anthropomorphism Perception: Developed from IDAQ scale, this 5-dimensional assessment system (emotional resonance, intentional attribution, mental simulation, role projection, social interaction) uses 7point Likert scales.

These scales were chosen for their cross-cultural adaptability and established reliability in assessing emotional states and mental health.

5 Experiment Results &Analysis

5.1 Technical Performance

Baseline Comparison SVC Models

To evaluate the SaMoye model in PetVocalia, we compare it against several baseline SVC models trained on our datasets:

Figure 4. Specific case analysis of audio converted using the PetVocalia (SaMoye model), where the pet timbre is the target timbre, shows that the mel-spectrogram and formant of the converted audio closely resemble those of the animal timbre.

• SoVITS-Flow 8 : AFlowmodel-based SoVITS-SVC, using Contentvec for representations.
• GPT-Sovits 9 : Originally a TTS model, adapted for SVC by replacing phoneme with F0 embeddings and training on our datasets.
• RVC [Kamble et al., 2023] 10 : Retrieval-based Voice Conversion (RVC) is a popular open-source, VITSbased SVC technique using retrieval of similar acoustic features.
• whisper-vits-SVC [Ning et al., 2023] 11 : Combines OpenAI's Whisper for speaker-independent content feature extraction and VITS for singing voice synthesis.
• DiffSVC [Liu et al., ]: A Denoising Diffusion Probabilistic Model (DDPM)-based SVC system, using Phonetic PosteriorGrams (PPGs) as content features and recovering target mel-spectrograms via a diffusion process.
• CoMoSVC [Lu et al., 2024]: A Consistency Modelbased SVC method via distillation from a pre-trained diffusion model (teacher) for few-step inference, significantly improving speed while maintaining performance.

8 https://github.com/svc-develop-team/so-vits-svc

9 https://github.com/RVC-Boss/GPT-SoVITS

10 https://github.com/RVC-Project/Retrieval-based-Voice-

Conversion-WebUI

11 https://github.com/PlayVoice/whisper-vits-svc

Zero-Shot SVC Benchmark Testing

As shown in Table 2, we found that compared to using human singers as the target timbre, using animals as the target timbre leads to a decline in model performance. This is because animal timbres are unseen data for all SVC models, representing an extreme condition for zero-shot singing voice conversion. Furthermore, the results show that SaMoye demonstrates significant performance improvements on both Human and Pet timbres compared to whisper-vits-SVC. Although its performance on Human timbres still lags behind CoMoSVC and DiffSVC, SaMoye's zero-shot performance on Pet timbres surpasses other models, thus making it more suitable as the base model for the PetVocalia functionality. This further explains that SaMoye's retrieval strategy during inference can fill information gaps in speaker embeddings, especially for animal cases where the timbre information gap is larger.

Ablation Study: Evaluation of Unfreezing the Speaker Encoder

We also evaluate the effect of unfreezing the speaker encoder during training. The results, as shown in Table 3, indicate that unfreezing the speaker encoder during training enhances performance across all metrics.

In addition, we use t-SNE to reduce the speaker embedding to two dimensions for visualization, as shown in Figure 5. We selected 328 speakers from speech datasets including Aishell, KSS, and VCTK (denoted in red), and 6,299 speakers from our singing datasets (denoted in purple). The figure illustrates that unfreezing the speaker encoder during training allows for a clearer distinction between speaker embeddings from speech and singing. This helps to enhance the timbre features of singing voices and improve the performance of zero-shot SVC tasks. This is because the wider pitch range in singing leads to more complex timbres compared to speech timbres, which typically have a very narrow pitch range. If not differentiated, using speech timbres for singing can lead to adverse consequences, such as intermittent sound issues in high and low pitch regions.

(b) Unfrozen speaker encoder.

Table 4. Experimental Results of Human-Pet Co-Creation. The meanings corresponding to each metric: PANAS-X (Pleasure), STAI (Anxiety), IDAQ (Anthropomorphism). Anxiety reduction scores represent change values (higher=better). Bold values denote optimal paradigm performance. Standard deviations in parentheses. All p<0.05.

5.2 Human-Pet Co-Creation Experiment Results

The experiment is implemented through the PetCoCre system, investigating a novel form of animal-assisted therapy involving collaborative music creation between humans and pets.

Main Effect (Co-creation vs. Solo)

In the first experiment, we aim to investigate whether higher satisfaction is exhibited when collaborating with pets in artistic creation compared to independent AI music composition. To validate this hypothesis, the independent creation mode of the control group (Solo Creation Group) allows participants to freely select digital instruments as rhythm/lead instruments and extract personal vocal timbre as lead vocals, without utilizing pet sounds as core elements. In contrast, the cocreation conditions of the experimental group (Co-Creation Group) require participants to use their pets' vocal characteristics (or a standard sample library) for AI music generation.

Experimental results, as shown in Table 4, reveal that artistic co-creation with pets significantly enhances users' creative pleasure and anxiety reduction effects compared to solo AI music creation. These findings provide empirical support for the digital transformation in animal-assisted therapy.

Paradigms Comparison (Within Co-Creation Group)

Furthermore, within the Co-Creation Group, we also systematically compared the differences among three humanpet music co-creation paradigms: PetRhythm (rhythm paradigm), PetMelody (melody paradigm), and PetVocalia (vocal paradigm). Each co-creation paradigm fundamentally incorporates pet sounds to enhance emotional engagement and explore their impact on the creative process.

As shown in Table 4, among the three paradigms, PetRhythm exhibited optimal performance in positive affect arousal and anxiety reduction, potentially due to rhythm synchronization-induced motor empathy effects, whereas PetVocalia achieved higher scores in anthropomorphism perception.

6 Conclusion &Discussion

This study introduces the PetCoCre system and the SaMoye model, which effectively integrates pet vocalizations into music creation by addressing the limitations of Zero-shot SVC open-source models. This process, utilizing pet vocalizations as their digital avatars, enhances emotional bonds and elevates the creative experience. Furthermore, this work pioneers a new paradigm for animal-assisted therapy, demonstrating significant therapeutic benefits in alleviating anxiety and enhancing pleasure. Moreover, this innovative approach highlights the potential of artistic co-creation in mental health interventions.

Ethical Statement

This study primarily focuses on human experiences and agency, emphasizing human emotional changes and anxiety alleviation during creation, while neglecting the active participation and experiences of animals. The current perspective only treats animals as passive participants, limiting the breadth of the overall definition of human-pet co-creation. Future research should aim to develop a more equitable cocreation system that allows animals to actively engage and express their emotions, thereby improving animal welfare and enhancing mutual understanding in human-pet interactions.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No.62272409).

Contribution Statement

Authors Zihao Wang and Le Ma contributed equally to this work.