MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarization

1 Introduction

Medical text summarization is useful for many real-life usecases such as summary generation of clinical records [Kanwal and Rizzo, 2022], health-related queries [He et al., 2021], and radiology reports [Dai et al., 2021]. Most medical summarization approaches are based on pre-trained language models (PLMs) that are trained on text from open-domain sources. Thus, their performance is sub-optimal because they did not incorporate medical knowledge into their models [He et al., 2021; Zhang et al., 2021]. Domain adaptation approaches for summarization in general [Fabbri et al., 2021a; Laskar et al., 2022; Xu et al., 2023] and medical summarization [He et al., 2021; Lamproudis. et al., 2022; Xie et al., 2023b], in particular, have garnered reasonable research interest. However, we identified two prominent research gaps in existing works.

We make the codebase and supplementary materials publicly available at https://github.com/gb-kgp/MEDVOC.

First, despite poor domain similarity of 33% in Figure 3(a) between CNN/DailyMail [See et al., 2017] (open domain) and PubMed Abstracts Collection (medical domain), none of the domain adaptation approaches for medical summarization update the PLM's vocabulary during fine-tuning. Figure 1 demonstrates the challenges that arise when generating medical summaries without vocabulary adaptation. Figure 3(b) demonstrates that medical concept-bearing words mostly lose their meaning due to poor representation because they are tokenized into four or more subwords. However, the undesirable tokenization actually happens at the decoder level during the generation of such medical conceptbearing words in a summary. Vocabulary adaptation is successful in the classification settings where it updates the PLM vocabulary adding a target domain-specific vocabulary [Hong et al., 2021; Xu et al., 2021; Lamproudis. et al., 2022]. Unfortunately, the vocabulary set construction algorithms of the classification setting are quite restrictive as they rely on fixed empirical thresholds [Tai et al., 2020; Hong et al., 2021] and fail to adapt themselves to a new, significantly different PLM architecture. In this paper, we are first to explore vocabulary adaptation techniques for summarization and design the MEDVOC fine-tuning strategy. However, adapting vocabulary adaptation strategies (earlier classification) to summarization (a generative setting) is non-trivial for MEDVOC: (i) the decoder needs to be additionally trained, (ii) rare sub-words get included in the target domain-specific vocabulary, and (iii) absence of large-scale medical summarization datasets for intermediate fine-tuning purposes. We address these challenges in this work.

The second research gap is that most of the vocabulary adaptation till now is evaluated on a single PLM, as shown in Table 1. To this end, as part of MEDVOC, we develop an efficient, dynamic vocabulary construction step that adapts to any encoder-decoder-based (PLM) summarization model and target (downstream task) datasets. We treat vocabulary construction as a hyperparameter tuning step and show that optimizing the fragment score of reference summaries in a given setting closely resembles the same optimization as the downstream task performance. This helps to avoid the extremely time-consuming step of intermediate fine-tuning. Our fragment score-based hyperparameter search very significantly reduces this fine-tuning time, from 450 days to ∼ 45 hours, averaged across three PLMs over four downstream medical summarization tasks.

To address the two key research gaps, we need to first train the PLMs on the downstream medical summarization tasks. However, the size of the training datasets of the downstream medical summarization task is quite small, in the range of 700 to 1525 data points. Directly fine-tuning pre-trained models on such small target datasets could lead to sub-optimal performance [Phang et al., 2019]. The standard approach relies on introducing an intermediate fine-tuning stage using large datasets [Chang and Lu, 2021; Suresh et al., 2023]. In this work, we show that the task of biomedical paper title generation serves as a good intermediate fine-tuning task [Fabbri et al., 2021a] for medical text summarization.

MEDVOC outperforms baselines by 15.74%, 4.80%, and 5.99% in zero-shot, few-shot, and full dataset settings respectively on average across four medical summarization datasets and three PLMs such as BertSumAbs [Liu and Lapata, 2019], BART [Lewis et al., 2020] and PEGASUS [Zhang et al.,

2020a] in terms of Rouge-L. MEDVOC produces more informative (5.99% Rouge-L improvement on average) and more faithful medical summaries (5.96% Concept Score improvement on average). We observe gains of 10.81% and 17.29% across challenging scenarios of long-form medical summary generation and reference summaries with high outof-vocabulary (OOV) concentration respectively. We make the codebase and supplementary materials publicly available at https://github.com/gb-kgp/MEDVOC.

2 RelatedWorks

To the best of our knowledge, this is the first work to explore vocabulary adaptation strategies for the summarization task. However, recent works have explored vocabulary adaption strategies for classification tasks and domain adaptation techniques for summarization, which we present in Table 1.

Vocabulary Adaptation Strategies For Classification. To

handle the vocabulary mismatch issue, BioBERT [Lee et al., 2019], retrained the model from scratch using a domainspecific corpus and showed performance improvement over the base pre-trained models. While other works like VOLT [Xu et al., 2021] and AVocaDo [Hong et al., 2021] aim to optimize the model's vocabulary by adding a set of subwords to the existing vocabulary using some utility scoring function. AVocaDo uses a fragment score-based [Rust et al., 2021] threshold, whereas T-DNA [Diao et al., 2021] selects n-grams with high Pointwise Mutual Information and iteratively merges them. exBERT [Tai et al., 2020] adopts an ad-hoc approach to determine the size of VTGT and fix it as 17K (56% of pre-trained vocabulary size), and does not perform any vocabulary optimization. However, all these vocabulary adaptation works have two major drawbacks: (i) they are limited to classification tasks, and (ii) they show results on a single model type and their algorithm is not flexible enough to handle different model types. To the best of our knowledge, this is the first work that explores vocabulary adaptation strategies for summarization over multiple PLMs. We focus on encoder-decoder-based PLMs because it is computationally infeasible to re-train medical LLMs [Chen et al., 2023; Wu et al., 2023].

3 ProposedMethodology

Here, we describe the MEDVOC fine-tuning strategy for adapting PLMs to medical text summarization tasks in Figure 2. We present the dynamic vocabulary construction step of MEDVOC in Section 3.1, and then explain the intermediate fine-tuning details in Section 3.2.

3.1 Dynamic Task-Aware Vocabulary Adaptation

The key challenge is that the base PLM's vocabulary (VPLM) remains unchanged during intermediate fine-tuning. Since this model vocabulary is obtained from training on opendomain data, we observe that VPLM misses (important) medical terms (Figure 3(b)) present in the target dataset, which occurs quite frequently across datasets considered in this study. This causes the PLM model tokenizer to split relevant medical terms into too many meaningless subwords, which ultimately is not able to capture the term semantics. This phenomenon is well-observed in prior studies w.r.t classification tasks [Xu et al., 2021; Hong et al., 2021] and results in poor downstream task performance. Therefore, we update VPLM by adding a set of target domain-specific subwords to the vocabulary, we refer to this updated model vocabulary as VMEDVOC.

Candidate Subwords Generation. We first identify a set of words present in reference summaries (RS) of target tasks that are poorly tokenized, i.e., split into more than 3 subwords. However, this resulted in poor coverage as it constituted a small fraction of 21.99% out of the total set of out-of-vocabulary (OOV) words on average, i.e., words split into more than one subword. Therefore, we also include medically relevant OOV words. We use the matcher.match function of QuickUMLS [Soldaini and Goharian, 2016] tool, with the parameters - (i) similarity measure as 'cosine', and (ii) similarity threshold at 95%, to obtain the set of such medically relevant words. We run the PLM's tokenization scheme on these selected words for the target task to obtain VTGT-TEMP. We apply the same procedure on the source documents of the Pubmed Abstracts Collection (PAC) dataset to obtain VPAC (see Section 4.4 for more details).

Vocabulary Construction Using Target Datasets (VTGT). We observe that VTGT-TEMP contain subwords that are infrequent in PAC and it is well known that adding infrequent downstream task-specific subwords may lead to the rare word problem [Schick and Sch¨utze, 2020; Hong et al., 2021]. Since PAC is used for intermediate fine-tuning, these infrequent words would appear in fewer contexts during training, and thus lead to a sub-optimal (noisier) representation. Therefore, we only consider subwords in VTGT-TEMP that overlap with the top K subwords of VPAC, thus mitigating the rare word issue. This value of K is the first hyperparameter for MEDVOC. However, we empirically observe that the size of VTGT is quite small as compared to the PLM's vocabulary size (3.66%, 1.58%, and 0.13% for BertSumAbs, BART, and PEGASUS respectively). This leads to marginal performance improvement as the added sub-words are overshadowed by the PLM vocabulary during summary generation.

Optimal Subset Selection From VPAC. Since this step is upper-bounded by the size of VPAC, which is quite large (3 times the model vocabulary), a large vocabulary causes parameter explosion and token sparsity problems, which hurts model learning [Allison et al., 2006; Xu et al., 2021]. Therefore, we put an upper limit equal to the |VPLM|. This also makes sure that the added vocabulary size does not exceed the PLM vocabulary size. Thus, determining the optimal subset size of VPAC lies in a tradeoff between model vocabulary size (large enough w.r.t PLM vocabulary) and model performance (small enough to not degrade downstream task performance). Therefore, we also include top P subwords from VPAC, where P = A× |VTGT|. The value of A thus forms the second hyperparameter for MEDVOC.

Efficient Hyperparameter Search Using Fragment Score. The standard hyperparameter search to find optimal values of A and K based on downstream task performance is extremely time-consuming. This is because it involves an additional intermediate fine-tuning step using PAC (312K data points) that takes 45 hours to run across 3 Tesla V100 32 GB GPUs (averaged across the three PLMs over four datasets). An exhaustive grid search over 240 settings takes 450 days. Our efficient fragment score-based hyperparameter search performs the same in 2 days on average, thus leading to a 240x speedup. Upon extensive evaluation, we observe that by optimizing the fragment score for a model type and dataset, the downstream task performance also gets optimized.

Algorithm 1: Vocabulary Construction of MEDVOC

1 Input: Pre-trained Vocabulary (VPLM), Model tokenizer type T , Source documents of Pubmed Abstracts Collection dPAC, Reference Summaries of Target dataset dTGT Output: MEDVOC vocabulary -VMEDVOC Initialization: K: Selecting Top-K subwords in VPAC, A: Factor over size of VTGT, Margin: 0.04 2 Function FragmentScore(dtrain, V ): 3 fC(V ) ← total count of subwords tokenized by V word count in dtrain 4 return fC(V ) 5 VTGT-TEMP ←CandidateSubwordsSelection (dTGT , T) 6 VPAC ←CandidateSubwordsSelection (dPAC, T) 7 for A← 0.25 to 10 by 0.25 do 8 for K ← 5000 to min(VPLM, VPAC) by 5000 do 9 VPAC-TEMP ← VPAC[0 : K] // Select top-K subwords 10 VTGT ← VTGT-TEMP ∩ VPAC-TEMP // Mitigating the rare word issue 11 P =min(VPLM,A · |VTGT|) // Size to sample from VPAC-TEMP 12 VPAC-TEMP-NEW ← VPAC[0 : P] // Select top-P subwords from VPAC 13 VMEDVOC-TEMP ← VPLM ∪ VTGT ∪ VPAC-TEMP-NEW 14 fragment score(A,K) ← FragmentScore(dTGT, VMEDVOC-TEMP)

15 minFragScore ←Minimum fragment score across all values of A, K

• 16 VMEDVOC ← VMEDVOC-TEMP with smallest vocabulary size with fragment score within a Margin of minFragScore

17 return VMEDVOC

Fragment score [Hong et al., 2021; Rust et al., 2021] is defined as the average number of subwords that a word gets tokenized into by a base model's tokenizer. We consider only those words which are split into more than one subword for fragment score computation on the target dataset. Since computing the fragment score requires the downstream task's reference summaries and is independent of the intermediate finetuning step, the time taken for hyperparameter search drastically reduces to only a few hours. We find that the lowest time taken is 1hour and 30 minutes on a single core of Intel i5 12core CPU for BertSumAbs on the CHQSum dataset, whereas the highest time taken is 5 hours and 45 minutes for PEGASUS on the EBM dataset. Therefore, the speedup observed by MEDVOC is proportional to the grid search space used for hyperparameter optimization. Our final vocabulary VMEDVOC comprises subwords that lead to fragment scores within a certain range of best-achievable (minimum) fragment scores on the target dataset's reference summaries. Algorithm 1 further explains the dynamic vocabulary construction step.

Time Complexity Of MEDVOC. The candidate subword selection step as well as the fragment score computation is proportional to the size of the input corpus (dPAC and dTGT). The hyperparameter search space is dependent on a constant set of values of A, while K is conditioned on |VPAC|, with the upper limit being |VPLM|, which results in time complexity of O(|VPLM| ∗ |dTGT|). The time complexity of MEDVOC is O(|dPAC|) + O(|dTGT|) + O(|VPLM| ∗ |dTGT|). Since, our target task datasets have limited size (|dTGT| ranges between 700 to 1525 documents) and |dTGT| << |dPAC|. The final time complexity is O(|VPLM| ∗ |dTGT|) + O(|dPAC|).

3.2 Intermediate Fine-TuningWith Biomedical Article Title Generation

Intermediate fine-tuning (IFT) is known to help PLMs when the downstream task has limited training (fine-tuning) data [Chang and Lu, 2021; Fabbri et al., 2021a]. In our case, the training dataset sizes range between 700 and 1525 data points (see Table 2), which is too small for training purposes and would easily overfit the PLMs, leading to poor performance. However, large-scale summarization datasets (similar to CNN/DailyMail in the open domain) are required for meaningful intermediate fine-tuning, which is absent in the medical domain. We show that biomedical article title generation satisfies the properties of a good intermediate fine-tuning task before fine-tuning on the downstream task of medical abstractive text summarization. Given that a good intermediate task aims to capture the knowledge or characteristics of the target task [Chang and Lu, 2021; Suresh et al., 2023], we use PAC as an intermediate finetuning task because it closely reflects key characteristics of the downstream summarization datasets (see Section 5.2).

Fine-tuning Details. Intermediate fine-tuning is performed using PAC (once for every PLM), whereas standard finetuning is done using downstream summarization tasks (once for every target dataset and PLM combination). Except for difference in the training dataset, we follow the standard finetuning procedure for the summarization task. We observe a marginal increase in the model's parameter count on adopting the MEDVOC strategy as only the embedding matrix corresponding to the added vocabulary needs to be additionally trained. The parameter count increments by 0.15%, 1.15% and 1.59% in case of PEGASUS, BART, and BertSumAbs.

4 Experimental Setup

We describe datasets and evaluation metrics, followed by baselines and training details (more details in Appendix A).

4.1 Target Task Datasets

We evaluate MEDVOC on two medical document summarization and two medical question summarization tasks. We provide detailed description of datasets in Appendix A.2.

Medical document summarization. In BioASQ [Tsatsaronis et al., 2015] and EBM [Moll´a and Santiago-Mart´ınez, 2011], each data point contains a query and PubMed (a biomedical database) abstract as the source document (SD) and an answer to the query as a reference summary (RS).

Medical question summarization. MeQSum [Ben Abacha and Demner-Fushman, 2019] and CHQSum [Yadav et al., 2022] contain consumer health questions posed by (nonmedical) users as the SD and a short question (manually curated by medical experts) as the RS. MeQSum and CHQSum comprise questions provided by the U.S National Library of Medicine and the Yahoo! Answers L6 corpus.

4.2 EvaluationMetrics

We report Rouge [Lin, 2004] and BertScore [Zhang et al., 2020b] (BSr) to assess general summarization quality. We use Rouge-L as the main evaluation metric in line with prior works [Yuan et al., 2022; Zhang et al., 2023a]. Concept F1score (CSr) is used to measure the faithfulness of medical summaries [Zhang et al., 2023b] and is computed as the overlap of UMLS medical concepts between the generated and reference summaries. We extract the medical concepts using the QuickUMLS [Soldaini and Goharian, 2016] tool. We also discuss an additional metric (MedRouge) and parameters of Rouge in Appendix A.3.

4.3 Baseline Models

We provide further implementation details for the baseline models in Appendix A.1 and A.4.

Vocabulary Adaptation Baseline Models. We obtain the BSA-PubMedBERT baseline by replacing the encoder of BertSumAbs (BSA) with PubmedBERT [Gu et al., 2021] respectively. In the same manner, we adapt a recent vocabulary adaptation model, AVocaDo [Hong et al., 2021] from classification to the summarization setting. AVocaDo selects subwords from a vocabulary and iteratively builds on a downstream dataset until the fragment score stays above a particular threshold (taken as 3).

Intermediate Fine-tuning Baseline Models. BioBART [Yuan et al., 2022] is obtained when BART is continuously pre-trained using PubMed abstracts corpora using only text-infilling as the pre-training objective. It achieves state-of-the-art performance on the MeQSum and CHQSum datasets. IFT-CNN describes existing PLMs that only perform intermediate fine-tuning with the CNN/DailyMail dataset. Unlike IFT-CNN, IFT-PAC uses the PubMed Abstracts Collection (PAC) dataset for intermediate fine-tuning. IFT-PAC is equivalent to MEDVOC without vocabulary adaptation. BSA-BioBERT is obtained similar to BSA-PubMedBERT where the encoder of BertSumAbs (BSA) is replaced with BioBERT [Lee et al., 2019].

4.4 Training Details

We obtain the biomedical paper abstracts from the official dump 1 of PubMed dated 2020. It comprises 450K data points (PubMed abstract and title). We refer to it as the PubMed Abstracts Collection (PAC) in the paper. We remove data points that overlap with downstream datasets (as EBM and BioASQ contain PubMed abstracts as the source document), as a decontamination step to prevent memorization issues [Radford et al., 2019]. We randomly select 312K data points to form the final dataset. We keep the dataset size of PAC similar to that of CNN/Dailymail for a fair comparison. Appendix A.5 provides further implementation details and the optimal hyperparameters and vocabulary size details.

5 Experimental Results

We show the performance comparison results of MEDVOC in Table 4. We observe an average Rouge-L improvement of 15.74% across datasets over baselines in a zero-shot setting. We further observe gains of 10.81% and 17.29% across challenging scenarios of long-form medical summary generation and reference summaries with high OOV concentration respectively. The consistent improvement also holds for Concept Score that captures the faithfulness aspect [Zhang et al., 2023b]. MEDVOC performs quite well for short-form summaries where it achieves SOTA performance on the MeQSum (even outperforming BioMedGPT [Zhang et al., 2023a]) and CHQSum data.

5.1 Performance Evaluation of MEDVOC

We investigate the MEDVOC performance using five research questions (RQs).

RQ1: MEDVOC outperforms vocabulary adaptation baselines. We observe that MEDVOC outperforms the vocabulary adaptation baselines of BSA-PubMedBERT and BSA-AVocaDo by a good margin of 33.10% and 3.94% respectively in terms of Rouge-L. We thus observe the effectiveness of designing the vocabulary adaptation as a hyperparameter tuning search (as explained in Section 3.1). We show consistent improvement due to the vocabulary adaptation step alone (MEDVOC versus IFT-PAC) across three different PLMs in the case of EBM and BioASQ, where MEDVOC improves over IFT-PAC by 3.55% and 5.38% respectively. We further observe that MEDVOC outperforms proportionately to the percentage of OOV words (higher the OOV%, MEDVOC outperforms more). We thus observe limited improvement in the case of MeQSum and CHQSum where the OOV percentage is 11.63% and 9.03%, which is much lower as compared to EBM and BioASQ where the OOV percentage is 14.67% and 18.46%. We extensively analyze the impact of the added vocabulary of MEDVOC and AVoCaDo in terms of fragment score in the Appendix B.1.

1 https://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles

RQ2: MEDVOC outperforms baselines even in zero and few-shot summarization tasks. We observe fromFigure 4(a) that MEDVOC consistently outperforms IFT-CNN across the full dataset, leading to improved zero-shot and fewshot (10 and 100-shot) abstractive summarization for CHQSum, MeQSum and EBM, with average performance gains of 28.94% and 8.13% in terms of Rouge-L, respectively. Remarkably, the advantage of MEDVOC is more pronounced in zero and few-shot settings compared to training on the entire dataset (15.74% versus 5.99%). In contrast, for the BioASQ dataset, IFT-CNN exhibits higher zero-shot performance than MEDVOC (Rouge-L score of 35.36), increasing to 42.31 in the full data setting (MEDVOC outperforms in full data setting by 8.67%) as reference summaries of BioASQ is extractive (characterized by unigram and bigram overlaps between SD and RS of 96.72% and 84.30%, respectively), similar to the well-known extractive nature of CNN/DailyMail dataset.

RQ3: MEDVOC outperforms baselines even when reference summaries have high OOV concentration. We select the top ten percentile of data points that have the highest OOV concentration in reference summaries. These points represent the most difficult data points in terms of vocabulary mismatch. MEDVOC shows a high improvement of 17.29% on average over IFT-CNN in the BioASQ and EBM

datasets that have an average OOV concentration of 41.95% in the selected data points. Figure 4(b) shows that higher the OOV concentration, higher the performance improvement as the PEGASUS model type with BioASQ dataset shows the highest performance jump of 40.53% over IFT-CNN.

RQ4: MEDVOC outperforms baselines in case of longer reference summaries. Generating long-formmedical summaries is under-explored [Liu et al., 2023]. Here, we select data points where RS length is greater than 30 tokens (which is the 95 percentile of length of summaries for MeQSum and CHQSum) and limit our evaluation to EBM and BioASQ. We observe that across these two datasets and three model types, MEDVOC outperforms IFT-CNN by 10.81% on average, which is quite higher than 8.23% improvement on the entire dataset. Therefore, MEDVOC improves the generation of long-formmedical summaries (Figure 4(b)).

RQ5: MEDVOC trains the model decoder to incorporate more relevant medical words during generation and produce more faithful medical summaries. We observe that BertSumAbs shows the highest performance improvement (based on Rouge-L) on average of 6.26% due to the vocabulary addition step, among the three PLMs. This is because the number of subwords required to generate a medical concept-bearing word (fragment score) significantly reduced from 2.09 to 1.58, the highest drop of 25% as compared to BART and PEGASUS of 3.47% and 3.14% respectively. We further observe that the faithfulness aspect measured using Concept Score (CSr) in a medical context [Zhang et al., 2023b] improved significantly on average across EBM and BioASQ, by 8.72%, 6.76% respectively. This indicates that MEDVOC generates more faithful summaries. Furthermore, we observe that a large percentage of top-3 candidate beams in the BertSumAbs model type contains 85.92%, 85.37%, and 86.16% of newly added MEDVOC vocabulary in the case of first, second, and third candidate beam respectively. This further highlights the positive impact of vocabulary adaptation for medical abstractive summarization.

5.2 Ablation Analysis

We observe an average Rouge-L improvement of 29% for IFT-PAC when compared with the IFT-CNN model in the case of EBM and BioASQ, in the high OOV concentration in the RS setting (as done in RQ3), where PEGASUS model on BioASQ dataset shows the highest performance improvement of 64.51%, as also observed in RQ3.

Biomedical paper title generations serve as a good intermediate fine-tuning task for medical summarization. We observe the following characteristics of PAC that are similar to the downstream datasets. First, the source document length of EBM and BioASQ, of 276 and 505 words, is more similar to the PAC dataset (276 words) as compared to CNN/DailyMail (700 words). Second, in the case of MeQSum and CHQSum, the length of RS is 12 words on average, and it is almost the same as PAC (15 words) as compared to CNN/DailyMail (57 words). Third, the abstractive nature of summaries of PAC aligns more with the downstream datasets. PAC, EBM, and MeQSum have a bigram overlap between SD and RS of 33.33%, 15.39% and 10.10%, respectively.

Biomedical paper title generation successfully aligns the positional embeddings to the target (medical) domain. The most informative part of a PubMed abstract is predominantly located in the Conclusion and Results section [Moll´a and Santiago-Mart´ınez, 2011; Jin et al., 2019], which naturally makes these sections more probable and desirable to be a part of a reference summary. Therefore, we observe in Figure 5(a) that the positional embedding shift due to IFTCNN over occurs the most at the start, whereas a consistently higher shift is seen for IFT-PAC after the initial around fifty tokens and the highest shift occurs towards the end of a SD of a PubMed abstract (token index ≥ 400). Thus, the domain mismatch is well-captured by positional embedding shift.

5.3 Human Evaluation

We randomly select 100 test data points uniformly across the four datasets and use the Prolific platform to recruit medical experts (self-reported to have a Masters or Doctoral degree in Medicine and Biomedical Sciences and are older than 24 years) for annotating summary pairs of MEDVOC and IFT-CNN across the standard aspects [Fabbri et al., 2021b; Zhang et al., 2023b] of relevance, coherence (on a Likert scale of 1 to 5), and faithfulness (binary). Each annotator was given 30 minutes to evaluate 10 summaries and was compensated at a rate of 9 UK pounds per hour (see Appendix B.2 for more details). Figure 5(b) shows the human evaluation results where MEDVOC generates more faithful summaries (88% versus 59% of summaries are faithful), and more relevant summaries, where 76% of data points get a positive score of 4 or 5 in Likert scale, as compared to 50% by IFT-CNN.

6 Conclusion

We present a dynamic vocabulary adaptation strategy, called MEDVOC, for fine-tuning PLMs for improved medical text summarization. To the best of our knowledge, this is the first work that uses vocabulary adaptation techniques for summarization and opens up an interesting potential research direction. Through extensive experimentation, we observe that MEDVOC consistently outperforms vocabulary adaptation baselines and significantly outperforms standard finetuning strategy (IFT-CNN) in full data setting. MEDVOC outperforms even in zero and few-shot settings, as well as when reference summaries have high OOV concentration or are very long (> 30 words). MEDVOC outperforms IFTCNN by a high margin in terms of relevance and faithfulness in human evaluation with medical experts. As an immediate future work, we will extend to the multi-document summarization setting. Given that MEDVOC leads to more faithful summaries, we will incorporate vocabulary adaptation to improve state-of-the-art models [Alambo et al., 2022; Zhang et al., 2023b; Xie et al., 2023a] that improve the factual consistency of summaries.

Acknowledgements

We thank the Ministry of Education, Govt of India, for supporting Gunjan Balde with Prime Minister Research Fellowship during his Ph.D. tenure. Soumyadeep Roy is supported by the Institute Ph.D. Fellowship at the Indian Institute of Technology Kharagpur. This research was partially funded by the Federal Ministry of Education and Research (BMBF), Germany under the project LeibnizKILabor with grant No. 01DD20003 and by a Google India Faculty Research Award.

Contribution Statement

Gunjan Balde and Soumyadeep Roy make equal contributions and share co-first authorship. Gunjan Balde is the corresponding author of this paper.