Temporally Aligning Long Audio Interviews with Questions: A Case Study in Multimodal Data Integration

1 Introduction

Audio surveys and oral interviews are routinely used as a means for data collection in many parts of the world [Jones, 2003], [Reichmann et al., 2010]. Apart from ease of use, audio surveys are also very inclusive since they naturally allow for data to be collected from illiterate or physically challenged individuals [Heinritz et al., 2022]. Several audio surveys by governmental and non-governmental organizations are conducted with the aim of improving social outcomes such as health and education. These surveys are typically accompanied by predefined questionnaires that guide the interviewer. Temporally aligning questions to where each question was asked in an interview becomes very important for organizations collecting the data, so as to quickly retrieve answers to relevant questions from long audio files.

In this work, we use long audio health surveys collected from mothers with young children residing in rural areas of Bihar, India. This data was collected by a non-governmental organization CARE India [CARE, 2023] as part of their larger effort towards improving maternal and neonatal health. The interviews are typically long, spanning more than 40 minutes on average, and are based on a fixed questionnaire designed by CARE. Given a written question drawn from the question-

the long recording that matches the given question. This task is of utmost importance to organizations such as CARE who would otherwise have to resort to tedious manual searches through these long audio recordings to retrieve answers to various questions.

Most prior work on cross-modal segment retrieval relies on strong supervision in the form of temporally-aligned text annotations [Anne Hendricks et al., 2017], [Gao et al., 2017], [Liu et al., 2018], [Xu et al., 2018], [Gupta et al., 2021], [Javed et al., 2022]. For long audio files, creating such timealigned annotations using human annotators would require significant amount of time and money. Since we are working with audio surveys derived from a fixed sequence of questions in a questionnaire, we use a weaker form of supervision involving larger audio segments paired with sets of temporally ordered questions. We present a new architecture INDENT 1 , that is trained using such weak supervision of larger audio segments and sets of questions. INDENT uses an attentionbased mechanism and is trained with contrastive losses across audio-text modalities to ground each question in the larger segment with its corresponding audio.

Our task of audio segment retrieval using textual queries is particularly challenging for the following reasons: 1. The audio surveys with the young mothers are conducted in noisy home environments, thus resulting in very noisy audio recordings. 2. Questions from the questionnaire are not asked verbatim and are typically paraphrased by the interviewer when posed to the mothers. Figure 1 illustrates this phenomenon. 3. While the questions appear in formal text in the official language of Bihar, i.e., Hindi, the audio recordings could contain variations that are typical of the local dialects spoken by the interviewees.

To summarize, our main contributions are: (i) We tackle a high-utility problem of locating short audio segments from within long audio surveys based on textual queries. These audio surveys are collected by CARE India in real environments and present many interesting challenges that we outline above. The dataset is enriched with fine-grained annotations on the evaluation splits and coarse-grained annotations on the training data that is sufficient to enable weak supervision. (ii) We design INDENT, a new model for text-to-audio retrieval from within long audio surveys given a question in text. INDENT is trained using weak supervision consisting of longer audio segments and sets of temporally ordered questions.

The problem of question-to-audio retrieval can also serve as an intermediate step for various downstream tasks such as audio-driven form filling and audio-driven question answering, by extracting answers from the survey recordings once the relevant question is isolated. We leave such extensions to future work and focus on the challenging problem of retrieving audio segments from long audios based on questions in text.

2 RelatedWork

Most relevant to our work are (i) approaches that focus on temporal sentence grounding in videos, where the task is to retrieve a video that semantically corresponds to a query in natural language, and (ii) audio text alignment for Automatic Speech Recognition (ASR).

2.1 VideoMoment Retrieval Using Text Queries

Prior work has explored various approaches for the task of temporally grounding captions or descriptions in videos in both fully and weakly supervised settings [Zhang et al., 2022]. Some of the prominent approaches are discussed below. Moment Context Networks (MCN) [Anne Hendricks et al., 2017] learn a shared embedding for video temporal context features (i.e., video features integrated with temporal endpoint features that indicate when a moment occurs in a video) and LSTM-based language features. The CTRL architecture [Gao et al., 2017] constitutes a temporal regression network to produce alignment scores and location offsets using combined representations for visual and text domains. A language-temporal attention network has also been proposed [Liu et al., 2018] to learn word attention based on the temporal context information in the video. Temporal boundary annotations for sentences have been used to train a segment proposal network using attention [Xu et al., 2018]. All these approaches assume the fully supervised setting. A Text-Guided Attention (TGA) model was proposed for video moment retrieval using text queries in a weakly supervised setting [Mithun et al., 2019]. However, TGA does not utilize the temporal order of sentences unlike our approach, INDENT that employs a Gaussian-weighted attention to leverage this temporal information.

1 INDENT can be expanded as "alIgning loNg auDio intErviews aNd quesTions".

2.2 Audio-Text Alignment For Speech Recognition

Most of the widely used ASR models that generate the best transcripts for various languages require large amounts of labeled transcripts for training [Chadha et al., 2022; Javed et al., 2022]. Methods proposed for low resource settings [Anguera et al., 2014] also expect access to some annotated transcripts during training, which are not available in our setting. Models that use CTC or sequence-to-sequence loss functions [Synnaeve et al., 2019; L´opez and Luque, 2022] require audio files with their corresponding transcripts to be used during training. Such models cannot be employed with our CARE India dataset, since we do not have any transcribed speech and the questions asked in the audio recordings are only semantically similar to (and often, paraphrases of) the questions listed in the questionnaire.

Recently, a few methods have been proposed for speech recognition using unaligned speech and text [Ao et al., 2021; Chen et al., 2022] and generating speech embeddings that capture the semantics of the underlying spoken text by aligning both text and speech embedding spaces using unsupervised techniques [Chung et al., 2018; Chen et al., 2018]. These models require word-level audio segments for semantically rich speech embeddings, which are generally not easily available or extractable. Other prior work require very large amounts of unpaired speech and text and do not currently serve many Indian languages [Ao et al., 2021; Chen et al., 2022].

3 CARE India Dataset

We create a new "CARE India Dataset" that contains speech interviews conducted by the CARE India Organization 2 . These interviews focus on analyzing maternal health and empowering women in rural areas in the state of Bihar, India. Each interview is approximately 40 minutes long and is recorded in the household premises of the interviewee, thus yielding very noisy recordings. The interviewer refers to a questionnaire as a guide for the duration of the interview. Searching for a specific question from the questionnaire within the noisy long audio recording is not the only challenge. Questions are not asked verbatim during the course of the conversation and could be paraphrased. The questions are also drawn from a diverse pool including multiplechoice type, numerical and subjective questions. All these factors add to the overall difficulty of isolating the correct audio segment that maps to a given question. CARE India (and other similar organizations) could greatly benefit from an automated solution that given a question in text can identify roughly where it appeared in a long audio recording. Without such a tool, volunteers would have to tediously go through long and noisy audio recordings to search for where a question appeared in it. This work is our first step towards building such a tool. We present a new model INDENT that can be trained on longer audio segments spanning multiple questions and identify potential audio segments within a test audio recording that could be matched with a test query.

2 https://bihar.care.org/

Each interview in the train set is annotated at the segmentlevel. A segment contains m consecutive questions and is of variable duration. Segments do not overlap or share questions. If a question begins within a segment, it would also end in that segment. We choose to set m = 5, since this granularity gives us segments that are not too long (which would make learning difficult) and not too short (which would otherwise overwhelm the human annotators). For each audio segment containing five questions, we annotate the start of the first question and the end of the fifth question. Our development and test sets have fine-grained annotations with start and end timestamps for each question. However, such a per-question annotation takes much longer. We find that annotating train files takes around 4 hours per audio recording, while it takes more than 6 hours to annotate test files from a similarly-sized audio recording. Detailed dataset statistics for our train, development and test sets are presented in Table 1.

4 INDENT: Architecture andMethodology

We formally define our problem as follows. Given a question q drawn from a fixed questionnaire Q and an audio interview f within which q is asked, we want to retrieve the audio segment s ∈ f that exactly matches q. On an average, f is more than 40 minutes long. In Table 1, we provide more detailed statistics on the data distribution. As described in Section 3, during training, we assume access to longer audio segments containing m = 5 questions. These segmentations are manually annotated. At test time, we extract non-overlapping audio segments of fixed duration from the test audio interview f and identify the segment that is the best match for a given question using our trained INDENT.

Overview of Workflow. INDENT consists of three main components: 1. Speech Encoder 2. Question Encoder 3. Gaussian-weighted Cross-attention Network. The speech encoder consists of audio preprocessing and feature extractor modules. We use an audio preprocessing module to remove noise from the speech interviews and break segments into smaller chunks of roughly 2 seconds duration each. We extract speech features at the chunk-level. The question encoder extracts question features at the sentence-level. Hence, the time resolution of the audio chunk features and the question features are very different (c.f. Figure 1). Using both audio chunk and question feature sequences as inputs, we try to align chunks with questions and simultaneously bridge the modality gap between speech and text in a shared space using a Gaussian Weighted Cross Attention module and contrastive learning. At inference, for a speech interview, we rank segments using the dot-product scores of the chunks and the given question. In Figure 2, we present a schematic diagram of the overall architecture of INDENT. Next, we elaborate on each of the model components.

4.1 Speech Encoder

The audio recordings contain speech both from the interviewer and the interviewee; the latter is typically very faint and more difficult to isolate. As mentioned in Section 3, the audio recordings are annotated with start and end times of large segments containing m = 5 questions each. Within each of these segments, we are only interested in parts corresponding to the interviewer. To isolate these parts, we attempted several techniques including voice activity detection (VAD) [Wiseman, 2017], speaker diarization [Bredin et al., 2020] and source separation techniques [Ravanelli et al., 2021]. After qualitatively analyzing the respective outputs, we found VAD to yield the best quality outputs. Using VAD, we split each annotated segment s into ns chunks {csi }ns i=1 . This results in a varying number of chunks across segments. We note here that the number of chunks ns can be higher than the number of questions m in a segment s. The microphone often fails to adequately capture the interviewee's voice, and we therefore assume (after qualitative validation)

that the chunks after VAD would primarily contain the interviewer's speech.

Once we have the chunk sequence {csi }ns i=1 , we use a frozen pretrained speech encoder (wav2vec2.0 [Baevski et al., 2020]) to extract features for chunks {csi }ns i=1 where csi ∈ RTi×d′ , Ti is the number of time frames for chunk csi . The speech encoder produces a d′ -dimensional feature vector for each speech frame in csi . To reduce csi to one aggregate feature vector per chunk, we apply a 1D convolution and take the mean across all Ti features. Next, we project these chunk features to a d-dimensional shared space using a linear layer. We then apply self-attention over the segment so that each chunk can gather context from other chunks. We also add skip connections wherever appropriate. The left block in Figure 2 represents the speech encoder with layers and skip connections. More formally:

where C s i ∈ Rd is the chunk feature after a Linear layer, SelfAttn(k, q, v) = softmax( k.q⊺ √d )v, q, k, v ∈ Rns×d and Csi is the chunk feature after self-attention.

4.2 Question Encoder

We use a frozen pretrained sentence transformer network to extract a d-dimensional feature vector. During training, for each segment s we have m questions, and hence we compute qs = [qs1 , · · · , qsm] where qsj ∈ Rd . We opt for sentencelevel features for textual questions since the spoken and textual questions are semantically similar at the sentence-level but need not have any correspondence at the word-level (e.g., when the spoken question is a paraphrase of the written question). Thus, extracting word-level features from the textual questions can lead to sub-optimal alignments. We also note here that we are doing weakly supervised training with no access to temporal boundaries but only the order of the occurrence of questions within a segment. This motivates our choice of using sentence-level features. The right block in Figure 2 shows the question encoder. It is important to keep the question encoder frozen since the pretrained sentence transformer is trained with massive amounts of text data in comparison to our training data. We only want to learn effective semantic alignments from the spoken utterances to the already pretrained sentence embeddings.

For each segment s, we now have speech chunk features Cs and textual features from the questions qs and we need to learn a cross-modal alignment between these two feature sequences. Towards this, we propose a Gaussian-weighted Cross Attention scheme.

4.3 Gaussian-weighted Cross Attention

Given Cs and qs , we aim to learn an alignment between qs , representing m sentence embeddings and Cs , representing ns chunk embeddings. We propose a Gaussian-weighted cross attention module with a contrastive learning objective

[Chen et al., 2020] in order to learn this alignment. Typically, ns > m implying that a question spans more than one speech chunk. To align a chunk Csi with a question qsi , we consider an "anchor chunk" feature ˆ Csi using qs . While chunk is anchored to one question, the neighboring questions can also provide context. We assume some consecutive chunks Csa:b combine to form the question qsi , but we do not know the temporal boundaries of qsi (given our weakly-supervised setting). Further, speech features Csi are not semantically rich to guide us with any weak boundaries. However, we know that each chunk unambiguously belongs to a single underlying question. Hence, we represent each chunk Csi as a linear combination of qs but anchor the chunk Csi to a single qsi by making one of the weights high and the rest much smaller to accumulate some neighboring context. A Gaussian distribution with a moving mean and standard deviation serves our purpose.

Start-of-segment chunks and end-of-segment chunks map to questions qs1 and qsm, respectively, with high probability. Hence, pivoting on ns, we move the Gaussian mean and vary the standard deviation as a function of the position of the chunk within the segment:

where µi and σi are the mean and standard deviation used to generate the Gaussian distribution Gi for chunk csi and α is a scaling factor. From Eqn 3, we see that for each position i, the mean shifts by m-1 ns-1 from the previously calculatedmean.

We sample values {aij }mj=1 from Gi as shown in Fig. 3, apply min-max normalization and generate an anchor chunk ˆ Csi using qs for each chunk Csi . Hence, {aij }mj=1 has a peak value aij and also has some weight allocated to the other questions. Thus, we generate a representation corresponding to each audio chunk Csi in the shared semantic space as ˆ Csi = ∑mj=1 aij qj s . Since ˆ Csi and Csi are both chunklevel representations, we apply a contrastive objective between them to bridge the modality gap. Since the question encoder is frozen, contrastive learning facilitates the speech encoder to better align with the text-based sentence embeddings.

Dynamic In-Audio Negative Sampling. For contrastive learning, we need negative chunks. Specifically, negative chunks from the same interview are desirable to prevent the model from learning easy solutions by merely exploiting speaker characteristics. During training, we create groups by randomly shuffling all segments in the speech interview and making groups of n segments. If the number of segments in the audio interview is not divisible by n, we also create a group with the last n segments in order to ensure that all the segments are covered during training. For each chunk in segment sx, the corresponding negative chunks will be from segment sy (x ̸= y) in the same group g. Chunks from all such sy are potential negatives for each chunk of sx. We denote the number of all potential negatives for sx by kx. To keep the number of negatives for each chunk constant inside a batch, we first compute the smallest kx across sx's in a batch and randomly sample those many chunks as negative examples for each chunk in the batch. We also explore data augmentation by performing reshuffling of all segments to create more new groups. The number of times we perform data augmentation is denoted by the hyperparameter D. Reshuffling is also performed across epochs.

Training Objective. We define the following contrastive loss [Chen et al., 2020] objective:

where ng is a union of i and dynamically sampled negative chunk indices (as described earlier), s, s′ ∈ g, s = s′ if j = i, C denotes all chunks in a batch; Lst aligns the speech and text spaces while Lss regularizes the network by explicitly encouraging negative speech chunks Csj (j ∈ ng, j ̸= i) to be far apart from Csi in the shared space. INDENT is optimized over the loss L = Lst + Lss. We apply the label smoothing technique [Reed et al., 2014] that forces INDENT to lower the confidence of the positive chunk by redistributing a small fraction of its probability mass uniformly to all the negative chunks. (In the experiments, we set positive probability to be 0.95 and redistribute 0.05 probability mass evenly among all the negatives.)

5 Experiments and Results

5.1 Experimental Setup

Question Encoder. We use pretrained Language-agnostic BERT Sentence Embedding (LaBSE) [Feng et al., 2022] as sentence embeddings. For each question q, LaBSE produces a 768-dimensional embedding. Since we keep the question encoder frozen, a helpful consequence of using LaBSE is that we can use questions in any language that are well-aligned in the LaBSE embedding space. In Table 2, we present experiments on questions in multiple Indian languages.

Speech Encoder. We use a pretrained voice activity detector (VAD) [Wiseman, 2017] to split segment s into ns chunks, {cs1 , · · · , csns }. To extract the chunkwise speech features ci s , we use a state-of-the-art multilingual model IndicWav2vecHindi [Javed et al., 2022]. Each chunkwise speech vector is a 1024-dimensional feature vector. To aggregate, we apply a convolution layer with 1 filter per channel (i.e., a total of 1024 filters) of receptive-field 20 and stride 2, followed by GELU activation [Hendrycks and Gimpel, 2016] and dropout and a final mean pooling on csi to derive ˆ csi . We then linearly project ˆ csi to a 768-dimensional vector to generate C s i ∈ C s . We subsequently apply self attention on C s to generate Cs . We present experiments with different speech feature extractors in Section 5.2.

Training. We train INDENT end-to-end keeping VAD [Wiseman, 2017], Question Encoder LaBSE [Feng et al., 2022] and speech feature extractor IndicWav2VecHindi [Javed et al., 2022] modules frozen using the loss defined in Section 4.3. We use the Adam optimizer with a learning rate 3e-4, a step scheduler with step-size 10 and decay factor gamma 0.1. After hyperparameter tuning (described in Section 5.2), we set batch size |B| = 4, n = 4, D = 2, σ = 0.5 and train for 40 epochs.

Model Variants. We refer to the complete model with the speech encoder and question encoder, trained using contrastive losses, as INDENT. To compare how INDENT performs with text instead of speech, we train INDENT-T by replacing the speech feature extractor in the speech encoder with ASR predictions for which we extracted LaBSE embeddings. NO-TRAIN is a learning-free technique that computes a simple matching based on dot-products between LaBSE embeddings of questions with LaBSE embeddings of ASR predictions from the speech.

Evaluation &Metrics. During inference, we create fixedchunk-size segments since we do not have segment boundaries during inference. We need segment information for local context; removing self-attention within a segment deteriorates the performance significantly. We set segment-size to 14 (average number of chunks in the train set). We compute dotproduct score q⊺Csi for the given question q across all chunks Csi . We rank each segment s using maxi=1···ns (q⊺Csi ) (higher score is better). We evaluate using the recall@k (R@k) metric. If the ground truth segment s of the question q appears in the top k highest scoring segments, the algorithm is considered to have retrieved the correct segment for R@k. Our reported R@k numbers are a mean across interviews of the ratio of the number of questions correctly retrieved to the total number of questions asked ( no. of correctly retrieved questions total no. of questions ). We report R@1, R@5, R@10 and average of the three is R-avg.

5.2 Results &Analysis

In Table 3, we present our main results. INDENT outperforms NO-TRAIN by an absolute 6.4% indicating that our training method has successfully aligned speech features to the semantic space of LaBSE embeddings [Feng et al., 2022]. INDENT is able to exploit very small amounts of weaklyannotated data to successfully bridge the modality gap and generate semantically-rich speech features. We also train INDENT-T using ASR transcripts instead of speech using our architecture and we see large (absolute) 26.4% and 19.98% gains in R-avg performance compared to NO-TRAIN and INDENT, respectively. This suggests: 1. The quality of ASR transcripts is reasonably good with WER 17.8 for the base model without any LM (as reported by [Javed et al., 2022]). 2. Aligning across speech and text modalities with small amounts of weakly labeled data is a challenging task. Bootstrapping using jointly trained speech and text models might be worth exploring as future work [Ao et al., 2021; Chen et al., 2022]. Our model INDENT still performs better than NO-TRAIN and can be quite effective when used with low-resource languages that do not have good ASR. We also perform experiments with other speech feature extractors such as Vakyansh [Chadha et al., 2022] and XLSR [Conneau et al., 2020], but both underperform compared to IndicWav2vec-Hindi [Javed et al., 2022].

A useful consequence of training INDENT using a frozen question encoder is that we can query our model with any of the 109 languages supported by LaBSE. We evaluate INDENT on 11 different Indian languages unseen during training. During inference, we translate the questionnaire using IndicTrans's Indic2English (for Hindi to English translation) and Indic2Indic (for Hindi to other Indic language translation) models [Ramesh et al., 2022]. Note that during training we use speech in Hindi matched to questions in Hindi, while during inference we query using text in multiple Indian languages. From the results shown in Table 2, we observe that INDENT outperforms NO-TRAIN for all Indian languages. INDENT-T performs the best, thus pointing to high-quality ASR transcriptions. Interestingly, performance on English is better with NO-TRAIN compared to INDENT. This could be because LaBSE was trained predominantly on English data.

Tables 4, 5, 6, and 7 show results on the dev set using INDENT from various ablation experiments by tuning important hyperparameters. Table 4 shows different group sizes. We see that having group sizes that are too large or too small are detrimental to performance. Note that each group contains an average of 14n chunks. We envisage that, with small groups there is not much diversity in negatives and with large groups a lot of potential negatives are never used since we only take minimum number of possible negatives across chunks in a batch (as described in Section 4.3). Hence, we train with n = 4.

Data augmentation hyperparameter D creates more groups of different segment combinations. Table 5 shows how data augmentation helps, but at the cost of overfitting. D = 3 leads to overfitting; D = 2 gives the best trade-off between train and dev set performance.

In Table 6, we present experiments with varying batch sizes. Similar to group size, we do not want the batches to be too large or too small. Although negative sampling is within a group, the number of negatives are the same within the batch and hence similar restrictions hold as with group sizes. Based on the results in Table 6, we set batch size |B| = 4.

In Table 7, we show experiments on tuning the standard deviation σ of the Gaussian distribution from equation 3. Using equation 2, we vary σ and scale it with α. In Table 7, we present the metrics across various α values and constant σ values. Our intuition behind using high σ in the center was that central chunks are hard to anchor, but Table 7 suggests otherwise. We see that constant σ is more effective by a significant margin, indicating that even central chunks are strongly anchored to some question.

Analysis. We qualitatively analyze INDENT by plotting all the chunk scores for a given question. As shown in Figure 4, INDENT is able to correctly identify the ground truth segment (in black); the dot-product scores are also very high for the ground truth segment compared to the rest.

6 Conclusions and FutureWork

We present a new end-to-end weakly-supervised approach INDENT, to align audio recordings with questions and an associated dataset of long health audio surveys collected from young mothers residing in rural areas of Bihar (India). INDENT automatically grounds questions within long audio recordings with the use of a Gaussian-weighted crossattention mechanism that exploits the fact that questions appear temporally ordered in the training audio segments. We show through extensive experiments that we are able to isolate questions within long audio recordings reasonably well and also demonstrate how this framework can be used with queries in other Indian languages.

Acknowledgments

Unlinked anonymized data used for the work reported here was collected by CARE India Solutions for Sustainable Development, as part of system strengthening and governmental program implementation tracking and supported by Bill and Melinda Gates Foundation. The technical and legal support was provided by the Koita Centre for Digital Health (KCDH), and specifically by Dr Raghavendran Lakshmi Narayanan, the Health Information Manager at KCDH. Shubham Nemani and Pranjal Saini's invaluable contributions and assistance during the initial data processing stages are deeply appreciated.