TRIKOP: Exploring Visual Prompting Paradigms for Multi-Grade Knee Osteoarthritis Classification on MRI Images

1 Introduction

Knee osteoarthritis (KOA) is a prevalent degenerative condition affecting the knee joint and is the most common form of osteoarthritis [Prieto-Alhambra et al., 2014], significantly reducing quality of life. The rapid advancement of Artificial Intelligence (AI) in healthcare has unlocked immense potential for analyzing medical imaging data and developing precise diagnostic systems. Vision Transformers (ViTs) [Dosovitskiy et al., 2021], renowned for their ability to capture global image relationships through self-attention [Vaswani et al., 2017], excel in computer vision tasks when trained on large datasets. However, their application in medical imaging remains challenging due to the scarcity of labeled data and the distinct characteristics of medical images. Transfer learning with pretrained ViT models, particularly through prompting methods like Visual Prompt Tuning (VPT) [Jia et al., 2022], has emerged as an effective adaptation approach. VPT employs learnable soft prompts for task-specific tuning, but its reliance on large datasets can limit its applicability in the medical domain, where data availability is often constrained.

∗Corresponding author

To address these limitations, we propose using domainspecific medical knowledge to design hard visual prompts that better guide pretrained ViTs. Rather than learning prompts from limited data, our method embeds meaningful priors to improve performance. We introduce TRIKOP-a trilogy of visual prompting strategies combining anomaly detection, GradCAM heatmaps, and contrastive learning-for KOA severity classification on MRI with frozen ViTs.

Our contributions are as follows. We propose TRIKOP with three novel hard visual prompting paradigms designed specifically for KOA severity classification, leveraging domain-specific medical knowledge to guide pretrained ViT effectively. Additionally, we develop a web application 1 2 that allows users to upload MRI scans, evaluate the proposed prompting methods, and visualize their outputs.

2 RelatedWork

2.1 Knee Osteoarthritis Diagnosis

Numerous studies have applied AI to diagnose KOA using medical imaging. For X-ray images, [Antony et al., 2017] employed CNNs and [Tiulpin et al., 2018] leveraged Siamese networks to classify disease severity. Transformer-based models have also been explored, with [Wang et al., 2023] applying ViT and modifying the original Positional Embedding of ViT to enhance feature extraction and improve performance.

MRI, offering richer structural details, has increasingly been adopted. [Guida et al., 2021] demonstrated that 3D CNNs outperform 2D X-rays in assessing severity. [Schiratti et al., 2021] used EfficientNet-B0 [Tan and Le, 2019] and Attention mechanisms to weigh the importance of MRI slices. [Berrimi et al., 2024] proposed a 3D CNN combined with Global Average Pooling and Residual Connections, improving feature extraction and prediction accuracy.

1 https://gmedai.com/app/main/mri-diagnosis

2 https://github.com/GlobalAILab/TRIKOP

2.2 Visual Prompting

The use of large pretrained models on extensive datasets for specific tasks is increasingly popular. Inspired by text prompting in NLP, [Bahng et al., 2022] and [Jia et al., 2022] explored visual prompting for vision models, demonstrating its potential in vision tasks. This approaches has since paved the way for promising applications in medical imaging. Medical Visual Prompting (MVP) [Chen et al., 2024] utilizes patch embeddings and superpixels for disease-specific segmentation, while DVPT [He et al., 2025] extracts sample and domain-specific features to reduce domain gaps with minimal trainable parameters. Similarly, [Denner et al., 2025] employs visual prompt engineering to enhance BiomedCLIP [Zhang et al., 2025], embedding visual cues in radiographs to improve focus on critical regions.

3 The TRIKOP Framework

In this section, we introduce the TRIKOP framework, which leverages visual prompting to efficiently and effectively adapt pretrained vision transformers for KOA diagnosis.

3.1 OverviewMethodology

The overall framework of TRIKOP is illustrated in Fig. 1, consisting of three main components: a frozen ViT backbone, a prompt module to generate prompt tokens, and a classification head to predict the severity of KOA.

The input 3D MRI volume I ∈ RD×H×W is divided into m fixed-sized patches {Ij ∈ RD′×H′×W ′ | j = 1..m} and embedded into a d-dimensional latent space with positional encoding, as in the original ViT. Simultaneously, input I is passed through a prompt module consisting of a prompt generator to create medical-context-aware prompt based on the input I and a prompt encoder to encode them into embeddings P = {P j ∈ Rd | j = 1, 2, . . . , p}, where p is the number of generated embeddings. These embeddings are fed into transformer layers along with image patch embeddings and the classification token ([CLS])'s embedding, guiding the model's attention to relevant features. This process can be expressed as:

[xi, ,Ei] = Li([xi-1, Pi-1,Ei-1]), i = 1..N (1) where Li is the i-th Transformer layer, xi-1 ∈ Rd is the [CLS] token embedding, Pi-1 ∈ Rp×d represents the prompt embeddings after passing P through the MLPi-1, and Ei-1 ∈ Rm×d is the image patch embeddings. After N transformer layers, the updated [CLS] token xN is passed through a fully connected layer for final KOA severity prediction.

To construct effective prompt embeddings P, we propose three paradigms for the prompt module: the anomaly map, the Grad-CAM heatmap, and the contrastive embedding paradigm.

3.2 Paradigm 1: AnomalyMap-based Prompting

In this paradigm, we leverage the Osteo-GAN model from our previous study [Phan et al., 2024] to generate anomaly maps using a reconstruction-based technique. Specifically, we pretrained an Osteo-GAN model on 50-th slices extracted from MRIs, as the 50-th slice represents the central region and generalizes well across knee MRI scans. This pretrained Osteo-GAN model serves as the prompt generator, as illustrated in Fig. 2.

Given a 3D input image I ∈ RD×H×W , the prompt generator GAno produces an anomaly map for the 50-th slice MAno = GAno(I50) ∈ RH×W .

The anomaly map MAno is then passed through the prompt encoder EncAno(·), which applies 2D patch embedding with positional encoding, similar to the embedding mechanism in ViT, to generate prompt embeddings P = EncAno(MAno) ∈ Rp×d .

3.3 Paradigm 2: GradCamHeatmap-based Prompting

Anomaly-based prompting, as a 2D prompt applied to 3D medical images, can introduce noise due to limited spatial representation. To address this, we propose Paradigm 2, which uses GradCAM [Selvaraju et al., 2017] to generate 3D heatmaps from a 3D CNN model. These heatmaps highlight key regions influencing the model's predictions, guiding the ViT to focus on medically relevant areas and improving classification performance. Fig. 3 illustrates this paradigm, which uses GradCAM heatmaps as prompts.

Firstly, we pretrained a supervised simple 3D CNN on the MRI dataset with a three-classification task (healthy (KL-0), early-stage (KL-1 & KL-2), advanced (KL-3 & KL-4)). This pretrained 3D CNN model is then employed as the generator GCAM in the prompt module. Given a 3D input image I ∈ RD×H×W , the prompt generator GCAM generates a GradCAM-based heatmap MCAM = GCAM(I) ∈ RD×H×W .

Subsequently, the prompt encoder EncCAM(·), which applies 3D patch embedding followed by positional encoding (similar to the embedding layer of ViT), encodes the heatmap into prompt embeddings P = EncCAM(MCAM) ∈ Rp×d in the TRIKOP framework.

3.4 Paradigm 3: Contrastive Learning-based Prompting

While Grad-CAM-based prompting enhances performance with detailed spatial information, it doubles the token count, significantly increasing self-attention costs in the transformer. To address this, Paradigm 3 reduces token complexity while retaining essential feature representations. Fig. 4 illustrates this paradigm, which employs contrastive features as prompts.

We first pretrain a 3D CNN model, referred to as the prompt generator GSupCon, using supervised contrastive learning [Khosla et al., 2020] to capture representations of disease severity for prompt generation.

The input image I ∈ RD×H×W is first processed by GSupCon, yielding a feature map MSupCon = GSupCon(I) ∈ R D 15 × H 32 × W32 , which encodes key discriminative features. This feature map is then flattened into a contrastive embedding PSupCon. Finally, PSupCon is projected into the d-dimensional space by a linear projection layer EncSupCon(·), generating the final prompt embeddings P = EncSupCon(PSupCon) ∈ Rp×d .

4 Experiment and Results

We utilized 3D DESS MRI sequences from the Osteoarthritis Initiative (OAI) project [National Institute of Mental Health, 2001]. Following the preprocessing pipeline proposed by [Guida et al., 2021], we modified the MRI volumes from their original resolution of 160×384×384 to 120×160×160 by cropping and removing non-informative slices. Instead of using center cropping as in [Guida et al., 2021], we applied automated cropping based on segmentation results to precisely capture the region containing the cartilage between the two bones.

#Params refers to the number of trainable parameters (Million). † Results are taken from [Guida et al., 2021]

Table 1: Comparison of results between prompting methods and the baseline.

Table 1 presents the benchmark results of three paradigms compared to other approaches. The results demonstrate that our visual prompting method achieves competitive accuracy with full fine-tuning while requiring significantly fewer trainable parameters.

5 Conclusion

In this study, we introduced TRIKOP, an application that utilizes Visual Prompting to enhance the diagnostic outcomes of KOA. It also enables users to visualize the affected areas on the knee joint that the model focuses on to produce diagnostic results through attention maps. In the future, we plan to refine prompt methods to improve the model's diagnostic accuracy. We anticipate that this application will bring significant value to patients and doctors in the treatment of KOA.

Acknowledgments

This research is funded by Viet Nam National University Ho Chi Minh City (VNU-HCM) under grant number DS2025-20-05. We acknowledge Ho Chi Minh City University of Technology (HCMUT), VNU-HCM for supporting this study.

We sincerely thank Prof. Truyen The Tran, Head of AI, Health & Science at Deakin University, for his insightful suggestion on applying visual prompting to KOA diagnosis. His valuable input not only inspired this research but also provided constructive feedback that further refined both the study and manuscript.