PillGood: Automated and Interactive Pill Dispenser Using Facial Recognition for Safe and Personalized Medication

1 Introduction

Dispensing correct medicines at the right time as prescribed for each patient is a task not only crucial, but also prone to mistakes in clinics [Wening and Breitkreutz, 2011]. Small human error, such as mistakes in pill taking-time, in the quantity, or the content of the medicines, endangers the medication safety and even threatens patient's life. Even at the outside of hospital, for those on a complex medication plan, keeping track of their medication records by following strict time table is particularly difficult and overwhelming which requires guidance of caretakers.

∗Contact Author

†Corresponding Author

Recently, with the increasing popularity of AI-powered devices [Levisse et al., 2019] especially using a smart phone application [Mantowsky et al., 2021] as well as with the increasing concerns of medication safety, it is desirable to deploy AI models on pill dispenser hardware. Furthermore, in order to handle the requirements of multiple patients in real time using mobile app, there have been some efforts along this line. Several commercial companies, such as TOSHO in Japan [Palttala et al., 2013] and JVM in Korea [Huat et al., 2014], developed an automated drug-packing system which automates drug allocation, sealing and labeling tasks according to the prescriptions collected from different units of the hospital through a computer [Beobide-Teller´ıa et al., 2018]. However, the demanding tasks required the most of human's discernment, such as notifying each patient at exact pill-taking time and identifying each patient, cannot be automated in these systems. There are more recent works to assist in dividing and dispensing medicines, including the smart pillbox using id-recognition [Tsai et al., 2020], the automatic medicine dispenser using alarm system [Mukund and Srinath, 2012], and the authentication system to recognize face of patient and showing their prescription on web [Chaiyarab et al., 2021]. However, they are not fully automatized with missing either the software component handling personalization or the hardware component controlling dispensation.

To meet the need of fully automated and personalized pill dispenser system, we developed PillGood, an automated smart pill dispenser system combining dispenser hardware with on-device facial recognition engine deployed using Raspberry Pi [Maksimovi´c et al., 2014]. Our system is fully automated and consisted with several components including the AI-engine, the speaker, the dispenser hardware, and the mobile app. Initially, PillGood requires each patient upload their prescription data through mobile app. Then, the system notifies patients through mobile app and speaker when they need to take the medicine based on each patient's prescribing information in the system, and detect who the patient is using AI-driven face recognition engine through attached camera on dispenser [Arya and Tiwari, 2020]. Finally, the controller collect and distributes pills to each patient following personalized prescriptions. Once the dispensation is done, the system sends the log file to the server to keep track of historical medication record. With best of our knowledge, this is the first framework that integrate both mobile notification module and face recognition module directly with the dispenser controller hardware, aiming to perform all tasks in real-time with fully automated way using the single unified system, and with interactive way integrating with mobile app.

We tested the effectiveness of the system on 10 users, and our results show that PillGood enables highly accurate personalized pill dispensation with averaged classification of 94%, and averaged rating of 5.5 out of 6.0.

PillGood provides a novel, yet simple and robust machinelearning-based personalized medicine allocation system which is very promising to use in clinical practice, and can potentially guarantee safety of personalized medication, reducing traffics in hospital and significantly saves time and labors of both patients and medical professionals.

2 SystemArchitecture

The workflow of PillGood is shown in Figure 1. PillGood consists of three components, mobile engine, AI engine, and hardware. After registration, the user takes a picture of face and the medication schedule through the mobile app. To ensure users to get a notification when they need to take the medicine, PillGood sends a notification to the mobile app and provide voice notification through the speaker attached to the hardware. The AI engine embedded in the hardware recognizes user's face, and dispenses prescribed pills accordingly. The historical log including time of pill dispensation, names of pills, name of users is saved in the server, and the log of each user is sent to their mobile app for their record.

2.1 Mobile Engine: Mobile Application

Interactive control of personalized medication is done by mobile engine by letting users to upload their prescription, and by providing the historical records of pill dispensations through the app. We developed IOS mobile app using Swift [Kaczmarek et al., 2019] which users can interactively update their medication schedules, get a notification at their pill taking time accordingly, and access to the information of the historical medication records containing time, date, and the names of the pills taken. To ensure users to be certainly notified according to the prescription time table, PillGood sends notifications both to the mobile app and to the speaker attached on the dispenser with human voice, calling the user's name.

2.2 AI Engine: Face RecognitionModule

For face recognition module, we use Haar feature-based cascade classifiers [Viola and Jones, 2001; Wang, 2014; Lienhart et al., 2003] combined with AdaBoost algorithm [Schapire, 2013] in which we first collect training data containing positive samples and negative samples, and then a cascade function is trained using the collected trained data. Then, we pick the strong classifiers using AdaBoost by dividing learned features into stages to form cascade classifiers. Haar cascade classifier algorithm is implemented by python using openCV library and deployed in Raspberry pi [Maksimovi´c et al., 2014]. Raspberry pi access to server and download database whenever there comes any update, such as users upload or update their prescription schedules. Here, we use RPI 8MP CAMERA BOARD connecting to Raspberry pi which has resolution of 5 megapixel and 720p (60fps) and 640×480p (60/90 fps) video. Raspberry pi takes user video on-the-fly and identify the user's face when the user is recognized more than 3 consecutive seconds. Once the device successfully identify the user, it collects the target user's prescription information from the database, such as time and name of pills to take. Then, Raspberry pi controls the hardware to dispense specific pills.

2.3 Hardware: Pill Dispenser

Our hardware is designed to dispense 1) the specific type and 2) the exact amounts of pills according to the prescription of target user. As shown in Figure 2, our pill dispenser consists of 5 parts including filling tube, plate, inner frame, exit passage, and ground. We manufactured all parts using 3D printer with flexible TPE filaments. Each plates of the dispenser store different type of pills, such as Tylenol, VitaminC etc. Initially, users are required to fill pills to each floor of the plates through filling tube. Once the prescription information of the target user is obtained, Raspberry pi controls the central motor to rotate plate in a way that pills are aligned as the one line across the outer arc of the plate. Then, the controller opens the exit passage on a plate of target pill type, and count exact number of dispensed pill using infrared sensor. Proportional-integral-derivative (PID) is adopted for rotation velocity control of the central motor [Willis, 1999], and Pulse Width Modulation (PWM) is used for the opening and closing control of the exit passage [Barr, 2001].

3 Results and Demonstrations

3.1 Demonstration onMobile Application

To assist users, PillGood provides interactive control through mobile app. As shown in Figure 3, users can set an alarm on what types of pills, which days, and what specific time they want to take the pills through the mobile app. Then, the entered data syncs with the device in real time through the cloud. The historical information regarding to when and what pills were taken is logged, so users can also easily keep track of this information through the mobile app.

3.2 Face Recognition Results

We evaluate the accuracy of our face recognition module on 10 different users. For training, 3,000 profile images were collected for each user through the embedded camera on Raspberry pi system. In test time, a single target user shows their face to the camera for 10 seconds, taking 1,000 pictures. Then, the system takes user video on-the-fly and identify user's face when the user is identified more than 3 consecutive seconds. The face is classified using Haar cascade by taking maximum probability from predicted categorical distribution of across 10 classes. As the output, it predicts bounding box of the face with the classified id with percentage accuracy. As we test one user at an inference time, we used recall as the main evaluation metrics. (We only measure true positive and false negative among 1000 test images of one user.) As a result, our face recognition engine achieves 94.0% test accuracy averaged across 10 classes. Figure 4 shows the variation of recall of classifying single target user from rest of 9 classes by changing threshold of classification accuracy from 0 to 100 %. As shown, the classification achieved perfect recall below the classification sensitivity of 37.8%. Our results show that simple features learn by Haar cascade provides robust and accurate enough performance of classification in our PillGood use case.

Figure 4: Recall with various threshold of classifying target user.

3.3 Qualitative Study: Effectiveness of the System

We evaluated the effectiveness of the system and the app usability through a user study. We recruited 10 patients with different and complex prescription plans, and let them to rate the efficacy and usability of our system after using it for one week. We used a Likert-scale [Albaum, 1997] between 1 (they will never use it again) and 6 (very useful and will continue to use it), in order to prevent voting to middle-way. As the result, the average score was 5.5, where all gave 5 points or higher. We also interviewed with one senior physician and discovered that there are lots of demands for safe and automated medicine dispensation system, like PillGood, especially in nursing hospitals where many patients have issue of keeping track of their medication records. Results from our user study and interview strongly supports the potential efficacy and usability of PillGood in an clinical environment.

4 Conclusions

We propose PillGood, an automated and interactive pill dispenser system using a machine-learning-driven face recognition model that integrates both interactive mobile application and AI driven face recognition module directly with the dispenser hardware. Thanks to our system, patients and medical professionals can minimize risk of making potential mistakes on medication error. Evaluation results prove the effectiveness of our system.