SaGol: Using MiniGPT-4 to Generate Alt Text for Improving Image Accessibility

1 Introduction

Alternative (alt) text, a general method of web accessibility for people with visual impairments (PVI) users, is text or phrases that describe images on a website in an understandable way [James Edwards et al., 2021]. According to the W3C's Web Content Accessibility Guidelines 2.1, alt text "serves the equivalent purpose" of the non-text content [Kirkpatrick et al., 2018]. However, many content creators tend not to create alt text for their content because they forgot to write [Jung et al., 2022; James Edwards et al., 2021]. Therefore, it is important to generate alt text automatically.

Previous studies employed various methods to create alt text, including the human-based method, image-search-based method, and AI-based method. Several studies utilized human-based methods to create alt text [Zhang et al., 2022; Gleason et al., 2019]. For instance, Zhang et al. developed a tool to generate alt text for GIFs by crowdsourcing alt text written by several people and choosing the best [Zhang et al., 2022]. This method has the advantage of producing alt text easily understood by humans. Other studies searched the image for the alt text that had already been made and crawled the text [Guinness et al., 2018; Pareddy et al., 2019]. For instance, Guinness et al. developed a tool that generates alt text for users by searching for a given image to look for existing alt text [Guinness et al., 2018]. This method has the advantage of providing quality-assured alt text without generating a new alt text. Additionally, some studies employed artificial intelligence (AI) models to generate alt text [Cho and Kim, 2023; Gleason et al., 2020; Jobin et al., 2022]. For example, Jobin K. V. et al. developed a tool that creates alt text on lecture slides using an optical character recognition module [Jobin et al., 2022]. This method has the advantage of being cheaper than having a human-generated alt text. However, all three methods have the disadvantage of being unable to reduce costs and maintain quality simultaneously. Also, these methods did not use a large language model (LLM) to generate alt text In addition, little is known about how to develop and evaluate tools for generating alt text in smartphone images and searching those images to assist PVI users. For example, images captured or downloaded with Android smartphones often lack alt text. Also, accessing images via Talkback only offers date information, rendering them inaccessible to blind individuals on Android smartphones.

To address this gap, we developed SaGol with two major functions. First, SaGol generates detailed alt text as it helps PVI users understand images specifically [Jung et al., 2022]. Second, SaGol searches for images using the generated alt text. This is because existing studies did not create a tool which generates alt text for images on smartphones. With these two features, SaGol helps PVI users easily understand and select images on their smartphones.

2 SystemOverview

We developed SaGol, an AI-powered image alt text generation and search application, to improve image accessibility for PVI users. SaGol 1) uses an LLM, MiniGPT-4, to generate alt text of images, and 2) searches for images based on the generated alt text [Zhu et al., 2023] (see Figure 1). SaGol helps users access the images on their smartphones.

2.1 Generate Alt text

For PVI users who have a challenge understanding images on their smartphones, We created a tool to generate alt text for images. First, SaGol loads images from the user's smartphone and uploads them to the AI server. Second, SaGol sends the loaded images to the AI server. The AI server generates the alt text for each image by inputting the question "compare similarity of vectors" along with the image to the MiniGPT-4 model. Third, the AI server sends the generated alt text to SaGol. SaGol stores the alt text on the smartphone.

2.2 Search for Images

For PVI users facing a challenge of selecting images on smartphones, we offer image search by alt text (see Figure 1).

Tokenize the Alt Text SaGol uses the NLTK module [Loper and Bird, 2002] for alt text tokenization and creates a Term Frequency-Inverse Document Frequency (TFIDF) matrix from the tokenized alt text and the search term.

Compare Similarity of Vectors SaGol vectorizes the user's search term of the user into a vector regarding the TFIDF matrix. SaGol compares the vector with other alt text vectors in the TF-IDF matrix using cosine similarity. Alt text vectors with high similarity contain many words in the user's question, so the more likely it is to answer the user's question.

Select Images Based on the similarity comparing results, SaGol selects images with alt text similar to the search term. The results screen shows the user with images with the most similar alt text to the search term. The user is allowed to get a detailed alt text of the image in voice.

3 Demonstration

SaGol, a smartphone image search application, improves image accessibility for PVI users by generating alt text and searching images based on the generated alt text. In addition, SaGol is designed for use with TalkBack, an accessibility tool for VBI users. SaGol works as follows (see Figure 2). Users initiate SaGol by clicking "Upload Images," which triggers an AI model to generate alt text for images on their smartphone. This step is only required the first time. After alt text are generated and saved, users enter search terms via voice recognition or manual typing. After clicking the search button, SaGol displays the 10 most similar images based on the generated alt text. Users can select an image using accessibility tools or direct clicks, prompting SaGol to provide a detailed alt text. By listening to the alt text of the images in the search results, users can confirm that they have found the image they are looking for. Both voice and alt text are accessible via TalkBack. SaGol not only benefits PVI users but also extends support to those searching for alt text or performing smartphone image search.

Usage Scenario 1 James, a 32-year-old visually impaired novelist known for his intricate descriptions, was looking for a suitable image of a white bicycle for his new novel cover. To overcome his PVI, he used SaGol to perform the voice image search for "white bicycle." SaGol provided detailed alt text for each image, including elements, such as surrounding houses, dogs, and trees. This allowed James to gather information and moods associated with the images, effectively overcoming his PVI and achieving success in his work.

Usage Scenario 2 Aliyah, a 35-year-old blind teacher who supports students with special needs, shares images with parents to show their children's learning activities and achievements. In particular, she highlights the images of cultural events or school festivals. To search for the images she wants, Aliyah says "school events" in SaGol. The application then searches for and delivers detailed alt text of various cultural events or school festivals, allowing Aliyah to vividly share her students' activities with parents, despite her PVI.

4 SystemEvaluation

The evaluation of our system was based on the length of alt text and the accuracy of the search. To perform the evaluation, we used the COCO dataset, which is a commonly used image dataset for object detection, segmentation, and captioning [Veit et al., 2016]. We randomly selected 300 images without duplicates to create a test set. The selection process was based on a normal distribution derived from the label distribution of the COCO dataset. The test set was used as input for several models to generate alt text for the images. We then measured the length of alt text and the accuracy of the image search using our test set.

4.1 Alt Text Word Count

To measure the alt text word count, we used MiniGPT-4 and BLIP to describe our sampled image dataset [Li et al., 2022; Zhu et al., 2023]. BLIP, a high-performance VisionLanguage Pre-training model, excels in image-text tasks using noisy web data, while MiniGPT-4 demonstrates advanced multimodal capabilities by generating detailed alt text. The text generated by each model was statistically analyzed for length differences (see Table 1). MiniGPT-4 had an average word count of 110.13, compared to BLIP's 11.85. A t-test yielded a t-value of 33.86 and a p-value of 3.31 × 10-141 . Since the p-value was less than 0.05, we rejected the null hypothesis and confirmed that MiniGPT-4's average word count was greater than BLIP's average word count.

4.2 Image Search Accuracy

Based on the TF-IDF, we performed the image search of alt text generated by several object recognition models and LLMs and compared their accuracy to determine the alt text generation model for SaGol (see Table 2) [Chollet and others, 2015]. To perform the image search, we used 300 images from the COCO dataset mentioned in subsection 4.1. Alt text for these images was generated using four object recognition models and two vision LLMs. We used the same 300 images as our evaluation dataset, and the evaluation process included the following steps: 1) select images and labels from the dataset; 2) tokenize the generated alt text and the search term; 3) create a TF-IDF matrix using the tokens and the search term [Aizawa, 2003]; 4) identify the 10 images with the highest cosine similarity to the search term in the TF-IDF matrix; 5) count as "found" if any of the 10 images is identical to the image used in the search, for accuracy calculation; 6) repeat steps 1) through 5) for all 300 images.

For evaluation, we included four object detection models: VGG16, ResNet152, MobileNetV3Small, and InceptionV3Small with the first two models [Simonyan and Zisserman, 2015; He et al., 2016; Howard et al., 2017; Li et al., 2022]. MiniGPT-4 showed the highest accuracy (see Table 2). Out of 300 images, 206 images were found with MiniGPT-4's alt text, while 193 were found with BLIP's alt text.

5 Conclusion

In this study, we developed and evaluated a smartphone application that allows users to search and understand images on their smartphones. However, our study has several limitations. First, while we evaluated the application using 300 images from the COCO dataset, we did not assess the usability and the alt text with the user's images. Moreover, despite the fact that we utilized the MiniGPT-4 model to generate alt text, we only evaluated two LLMs to select a model for generating alt text. Additionally, although we performed the image search using the alt text generated by the models, we did not perform any accuracy enhancements, such as fine-tuning and setting a threshold for the search. Furthermore, we used the prompt "Describe the image" as the default in the model, but no evaluations were conducted with different prompts.

Accordingly, we propose the following recommendations for future research. First, we should assess the usability and the accuracy of the image search with PVI users to identify areas for improvement. Moreover, we need to evaluate other LLMs to find more appropriate ways of generating alt text, thereby improving SaGol's performance. Plus, we ought to improve the alt text and search accuracy by fine-tuning, setting thresholds and exchanging the model with the existing LLM API. In addition, it is important to provide different prompts to improve accessibility for PVI users of different ages, nationalities, genders and education levels. Our research aims to enhance image accessibility for PVI users by generating detailed and low-cost alt text through SaGol.