Building a Personalized Messaging System for Health Intervention in Underprivileged Regions Using Reinforcement Learning

1 Introduction

Non-communicable diseases (NCDs) such as cardiovascular disease, diabetes, and cancer, are among the top health challenges of the century. According to WHO, NCDs kill 41 million people each year, equivalent to 74% of all deaths globally. Notably, 85% of premature deaths from NCDs occur in low- and middle-income countries [Organization, 2018]. Fortunately, these serious diseases are largely preventable. According to WHO, NCDs are preventable with three lifestyle changes: eat healthy food, increase physical activity, and avoid tobacco. Yet barriers in underprivileged regions often prevent engagement in these activities, particularly poor education about the disease, lack of social support, and limited healthcare access.

We propose to overcome some of these barriers by building a new AI-based system which can help in improving diabetes risk behavior of people in such underprivileged regions. We build an effective messaging intervention system, that dynamically sends personalized messages to participants (through Whatsapp). These messages contain information about diabetes causes and complications, and the impact of nutrition and fitness on preventing diabetes. We chose Whatsapp to transmit our messages since mobile phone uptake is high in India, and Whatsapp is an essential communication channel that is accessible to almost everyone. Existing non-profit healthcare programs often pre-design fixed non-personalized messages [Pfammatter et al., 2016; Ranjani et al., 2020]. Our work seeks to improve this using AI, leveraging techniques in RL to optimize our message selection policy and tailoring sent messages to align with each individual participant's needs and preferences.

We provide four main contributions. First, we build an online diabetes-targeted intervention system that automatically sends out messages to participants in our study on a weekly basis. Each week, messages sent to each participant are determined based on behavior dynamics of the participant. In the same week, our system collects information about changes in behavior of participants through a question/answer mechanism, which is leveraged to improve our message generation in later weeks. Second, we model the problem of optimizing message generation as a RL problem and develop a RL algorithm for solving this problem, tackling concrete real-world challenges exhibited in our problem domain. Our algorithm is an extension of DQN [Mnih et al., 2015], a well-known RL method, with adaptations to handle simultaneous interactions with multiple participants in an online learning fashion.

Third, we run an extensive field study that involves over 1000 participants from local villages that received messages generated by our system for more than six months. Our analysis shows that there are significant improvements in the participants' diabetes-related knowledge, physical activities, and high-fat food avoidance at the end of our field study. Finally, we build a new neural net-based behavior model to predict participants' behavior changes, leveraging the data collected during our study. We show that our model, which exploits inherit differences in behavior changes across different types (knowledge, physical activity, and food consumption) obtains the best prediction accuracy compared to baselines.

2 RelatedWork

Reinforcement Learning in Healthcare. RL has been widely used in tackling various problems in healthcare. In particular, RL was used in developing effective personalized treatment plans which can be adaptive to the dynamic changes of clinical states. There are several works in this line of research including studies of chronic diseases such as cancers [Zhao et al., 2009; Ahn and Park, 2011; Hassani and others, 2010; Padmanabhan et al., 2017], diabetes [Bothe et al., 2013; Daskalaki et al., 2013; Noori et al., 2017; Asoh et al., 2013], anemia [Gaweda et al., 2005; Gaweda et al., 2006; Mart´ın-Guerrero et al., 2009], and HIV [Yu et al., 2019; Parbhoo et al., 2017]. In addition, there is an increasing number of studies that applied RL techniques to problems in critical care such as generating optimal sepsis treatment policies [Saria, 2018], and anesthesia control [Moore et al., 2014; Sinzinger and Moore, 2005]. RL was also used in automated medical diagnosis [Sahba et al., 2008; Ghesu et al., 2017; Chu et al., 2016; Kao et al., 2018]. We refer readers to [Yu et al., 2021] for a complete literature review.

The research topic that is closest to our work is the problem of health management. Specifically, there are works on using RL to optimize messages sent to users to improve their physical activities [Hochberg et al., 2016; Yom-Tov et al., 2017]. They essentially developed a mobile phone app that runs in the background of patients' smartphones and automatically collects data of physical activity performed by patients. They then run RL that utilizes the collected data to determine which SMS message is likely to increase the physical activity of the patient. This approach requires consensuses from patients to record all of their physical activities. Our work focuses on developing a personalized text message mechanism which targets not only physical activities but also diabetes-related knowledge and food consumption. This is accomplished through a automatic message/question/answer process in which participants can opt to respond or not.

Knowledge Tracing. Our behavior modeling of participants is related to knowledge tracing, an important research research areas for enhancing personalized education [Corbett and Anderson, 1994]. The main task is to build machine models of the knowledge of a student as they interact with coursework. Recently, given the rise of deep learning, there have been several works that utilized deep neural nets to model the student learning [Piech et al., 2015; Yeung and Yeung, 2018; Xiong et al., 2016; Ghosh et al., 2020; Pandey and Karypis, 2019]. Our work, on the other hand, focuses on building a ML model of behavior change of participants in our study as they interact with our intervention system. We target not only diabetes-related knowledge improvement of participants, but also their physical activity and food consumption dynamics, as influenced by our messages sent to them on a weekly basis.

3 PersonalizedMessage-Generation System

In this work, our goal is to optimize the impact of our message intervention program on participants' lifestyle behavior. The challenge is that participants have varied lifestyle and also may have various kinds of reaction to our messages. Therefore, it is important that we can personalize the message selection policy for each individual participant. We propose to apply techniques in reinforcement learning to serve this purpose. Overall, given a message bank as an input, our RLbased system runs for a number of rounds (i.e., weeks). In each round, our system selects a pair of messages for each participant and sends out these messages to the participants. In addition, the system sends out a separate pair of questions to each participant and collect answers from them. These questions collect information regarding the changes in the participants' lifestyle on a weekly basis. Our system then uses the participants' responses to these questions as feedback to update our message selection policy. The overview of our system is illustrated in Figure 1.

Our messaging process has three distinct phases. In phase one of participant sign-up, health workers visit villagers in person. The health workers (i) sign up people for receiving our health-related messages; (ii) provide detailed instructions of the program to the participant; and (iii) distribute initial questionnaire and collect answers from participants. The initial questionnaire includes questions about the participants' demographics, family health-related history, knowledge about diabetes, physical activities, and food intake, etc. We use participants' responses to this questionnaire to build the initial state of participants.

In more detail, each participant's state consists of "scores" for the five categories: (a) healthy food intake, (b) unhealthy food/tobacco/alcohol intake, (c) fitness/physical activity level, (d) diabetes cause knowledge, and (e) diabetes complication knowledge. The scores take value in {1, 2, 3} where 1 means "Low", 2 means "Medium", and 3 represents "High". For example, the state for a participant can be (1, 3, 2, 2, 3) where the score for the health food intake of this participant is 1, it means that this participant rarely consumes healthy food such as vegetables and fruits. Thus, this score implies that this participant should receive messages that encourage the participant to eat more healthy foods.

In phase two, participants receive two messages and two questions each week, each message or question targets one of the above five categories. The sets of messages and questions are carefully designed in local language by domain experts. In this phase, our message-based intervention system interacts with participants via Whatsapp. The flow of the weekly interactions with each participant is illustrated in Figure 2 and an example of weekly messages/questions is provided in Figure 3. As we show in Figure 3, the purpose of the questions asked in each week is to determine the impact of the messages sent out in the prior week on the participants' behavior change. We remark that our system sends out only two messages and two questions on two days per week (i.e., Tuesdays and Fridays and one message/question per day) for the sake of participants' comfort. Overloading the participants with messages/questions can potentially disrupt participants' daily activities, causing unnecessary burden, decreasing their engagement in our program. Finally, responses of participants to our questions is used to update the participants' state. Our system then leverages the participants' updated state to generate new messages and questions in the following week.

Finally, in phase three we perform a post-study analysis, in which our health workers meet participants in person again and conduct the same questionnaire as in phase one. Our goal is to have a complete comparison of the behavior changes in beneficiaries before and after participation in our program.

4 RL-basedMessage Generation Algorithm

The core of our message-based intervention system is the RL agent that actively selects weekly messages and questions for individual participants in a personalized manner based on their previous responses. We adopt ideas from Deep Q-Learning (DQN) [Mnih et al., 2015], a well-known RL method in literature, to build our RL agent. However, we face the following challenges. First, RL methods are effective typically when they are pre-trained before the actual real-world deployment. However, in this project, there is no historical participant data that can be used for pre-training RL methods. Our RL agent has to be trained directly on the job during limited weekly interactions across only 25 weeks. Second, ideally, we can treat each participant as a separate Markov Decision Process and train a separate RL agent for each participant. However, our system can only obtain a single trajectory of state transition for each participant, which is extremely limited information for building an RL agent for each individual. Conversely, a single RL agent for the whole population may fail to capture the diverse behavior of participants. Third, traditionally, RL models are iteratively trained by sequentially acquiring different episodes of interactions with the environment. However, here our system interacts with multiple participants in parallel across a single episode.

Given these real-world deployment challenges, we create a new variant of DQN with the following revisions. First, we provide a warm-up stage in which we "pre-train" our model; we leverage the participants' responses to the initial questionnaire to initialize values of learnable parameters of our model. Intuitively, these values are determined such that our model generates messages for each participant that target state categories in which the participant has low scores. For example, if the participant has low physical activity level (i.e., score is 1), the model will likely select a message that encourages the participant to do more physical exercise. Second, based on initial state scores, we use a clustering method to divide participants into three groups such that participants in the same group have similar initial states. We aim to train a single RL agent for each group, anticipating similar in-group behavior. Note that this means the RL agent is trained based on data collected from all participants within a group; however, messages and questions selected by the RL agent for each participant are determined by their individual state. By doing so, we use enough data to train our model while still personalizing messages. Third, we modify the DQN training process to allow multiple updates of model training in each step (a week) and simultaneous state and message updates for all participants. The details are presented in Algorithm 1, which runs separately for each participant group.

Essentially, in each week t, each participant i is associated with a state s(i) t-1 which represents the latest status of

Algorithm 1: Adaptive Message Generation.

the participant lifestyle behavior as we discussed previously in Section 3. The goal of Algorithm 1 is to train a neural net model with unknown parameters θ to predict the q-value Q(s(i) t-1 , a, θ) of the state-action pair (s(i) t-1 , a). Here, an action a represents a pair of messages selected from the weekly message bank. Intuitively, the q-value Q(s(i) t-1 , a, θ) is the total expected reward that we receive if the action a is chosen for participant i given the latest state value s(i) t-1 . This total expected reward captures the long-term impact of the selected action on the behavior change of participant i in the future.

Warm-up. In the warm-up phase, we pre-train our neural net model by minimizing the following MSE:

where init.value(s(i) 0 , a) is the estimated importance score of action a for participant i given the participant initial state is s(i) 0 . Note that this initial state is estimated based on the participant's response to the questionnaire. The score init.value(s(i) 0 , a) is determined such that this value will be high if the action a includes messages that target categories that the participant has low scores, and vice versa. For example, if the participant has score 1 for the physical activity category in s(i) 0 , the value of init.value(s(i) 0 , a) is 3 if the action a has physical activity-related messages. Here, N is the total of participants in the considering group and A is the set of all possible actions (i.e., pairs of messages).

Weekly message/question selection. In the weekly message/question phase, at every week t, for every participant i, with a probability of ϵ > 0, we select an action a(i) t for participant i uniformly at random. And with a probability of 1 - ϵ, we select the optimal action a(i) t = argmaxa Q(s(i) t-1 , a, θ). This ϵ-greedy approach allows us to balance between exploration and exploitation during the learning process. We then send a pair of questions q(i) t-1 . These questions are used to measure the impact of messages the participants received in the last week t - 1. We remark that if we receive responses from participants for these questions quickly, we can immediately use these responses to update participants' state and then generate newmessages for this week. However, this is not the case due to delayed updates, meaning that we have to send out messages for this week prior to receiving responses. Therefore, we have the following message/question procedure:

• Week t = 1: we send out messages a(i) 1 based on s(i) 0 .
• Week t = 2: we send out messages a(i) 2 based on state s(i) 1 = s(i) 0 . We then send out questions q(i) 1 regarding impact of a(i) 1 and receive responses answer(i) 1 . We update state s(i) 2 = Update(s(i) 1 , q(i) 1 , answer(i) 1 ).
• Week t = 3: we send out messages a(i) 3 based on state s(i) 2 . We send out questions q(i) 2 regarding impact of a(i) 2 and receive responses. We update state s(i) 3 = Update(s(i) 2 , q(i) 2 , answer(i) 2 ), and so on.

Note that, in reality, participants may not respond every week. When we do not receive any answer from a participant, there will be no state change for that participant, i.e., s(i) t = s(i) t-1 . Model training update. Similar to DQN, we maintain a replay buffer D of historical interactions with participants. At every step t, transitions (s(i) t-1 , a(i) t-1 , s(i) t , r(i) t-1) of every participant i will be added to D. Here, the reward r(i) t-1 is calculated based on the state change (s(i) t-1 , s(i) t ). For example, if the physical activity score part of the state is updated from a value of 1 in s(i) t-1 to a value of 2 in s(i) t , then the reward r(i) t-1 = 1. This reward value indicates that the action a(i) t-1 had positive impact on the participant i's exercise behavior. If multiple categories in the state have their scores changed, then the reward is computed as the sum of rewards over all these categories. This buffer D is used to update the neural net parameters θ. In addition to D, we also maintain a target network ̂Q with target parameters ̂ θ. The values of the parameters ̂ θ are updated periodically based on the network Q. The replay buffer D and target network ̂Q are the main ideas of DQN that help in stabilizing and improving the q-learning process. Finally, at each week t, after updating D with new transitions, we run a number of iterations numUpdate to perform gradient descent updates on θ based on the loss [rj +γmax a′ ̂Q (sj+1, a′ , ̂ θ) - Q (sj , aj , θ)]2 which is computed based on a mini-batch of transitions (sj , aj , sj+1, rj ) sampled from D.

5 Real-world Deployment

For real world deployment, our primary goals were to lay the groundwork for our message intervention program. These included completing all the required preparations, obtaining IRB approval, working with domain experts to design a bank of messages and questions, completing front-line worker training, collecting baseline data, and testing the transmission system. In addition, our critical objectives included recruiting beneficiaries and commencing the message transmission.

In January and February of 2022, we successfully set up the automated messaging pipeline by linking our Google Cloud VM to our partner's cloud storage, allowing for seamless and automatic transmission of messages and questions selected by our RL system to participants. We also completed Facebook business verification, obtained approval for WhatsApp message transmission, and developed the AI tool. Crucially, we ensured that key people from multiple project partners could seamlessly share de-identified data files and responses from the villagers each week. Finally, we completed training for 20 front-line workers on the project implementation.

In February and March of 2022, our front-line workers collected behavior surveys (questionnaire) from 1698 participants who are local villagers in India. We successfully completed a pilot test of the diabetes-related AI messaging bot to verify functionality of the overall system. In the end, we successfully recruited 1049 participants to opt-in to receive the diabetes messages. To compare with the existing static message program in which all participants received the same sequence of messages, we randomly divided participants into two groups: 548 participants joined our AI-based message program (we call this group the AI group) and 501 joined the baseline static message program (the non-AI group).

In March of 2022, we began the message transmission for the first batch of users with those in the AI cohort receiving two messages and two questions weekly on Fridays and Tuesdays. Participants in the non-AI group received two messages on Thursdays and Mondays each week. We remark that all participants did not opt-in at the very start but gradually joined over a couple of weeks. The villagers in the AI group responded to questions asked - engagement levels were around 35% past week 8 of the study.

The study ended in November 2022, at which time the post-study questionnaire was sent to the participants. This is the same questionnaire used at the beginning of our study. Based on responses of participants, we are able to evaluate the effectiveness of the AI system by comparing the performance of participants between the AI and non-AI groups.

6 Post-Study Intervention Result

We analyze behavior changes in both AI and non-AI participant groups based on their responses to the same questionnaire before/after joining our study. We divide questions into different types that focus on knowledge, physical activity, and dietary. Responses to each question are converted into three scores: 1 (low), 2 (medium), and 3 (high). The final score of a participant in each category is averaged over all questions in that category. We compute the score difference between the pre- and post-studies for each participant. Positive score changes imply participants improve their behavior at the end of our study. Our analysis results are shown in Figures 4, 5, 6 where the x-axis represents the participants and the y-axis represents the score change.

To compare our AI versus the non-AI intervention, we consider three statistics: (i) mean score change across each population; (ii) percentage of participants who improve their scores; and (iii) percentage of participants who decrease their scores. For example, in Figure 4 for the AI group, 58.3% of the participants have a positive score changes while 19.4% have a negative score changes in the diabetes-cause knowledge type. In addition, the mean score change is 0.387. For the AI group, we only consider participants who have response rates of at least 50%. Our rationale is that lowresponse rate participants do not engage in the study, and thus, their behavior will not be impacted by the AI messages.

7 Human Behavior Modeling

We aim to build a predictive model characterizing how participants behave in response to our message intervention program. Such predictive model could be used in the future to create simulated data to refine our message selection policy.

Feature Extraction. We use data collected from our field study for this modeling task. We divide the questionnaire and the weekly messages/questions into 17 finer categories. The questionnaire responses are then used to compute scores in each of these categories for all participants, which are then used as features. Additionally, we include the message and question ID that the participant received, along with the category that they belong to. To provide the model with more context, we include the previous week's information for all these categories, along with the responses received from participants. Lastly, we also include a coarser categorization feature, indicating whether the question asked is regarding one of three types: knowledge, physical activity, or dietary.

Model Description. The prediction task, then, is to use the extracted features to predict participants' responses to the questions they receive. We cast this problem as a multiclassification problem. In each week, for each participant, we take each question and other features associated with that participant as an input to produce a prediction of the corresponding response of that participant (which is categorized into three levels 1 (low), 2 (medium), and 3 (high)).

We consider three different models as baselines for this task: (i) the classic logistic regression; (ii) a simple neural network (NN) with two fully-connected linear layers; and (iii) a Long Short Term Memory (LSTM) based model [Hochreiter and Schmidhuber, 1996] (i.e., a LSTM block followed by a simple linear layer). We remark that LSTM is commonly used in deep knowledge tracing [Piech et al., 2015].

Figure 6: Dietary Comparison. We plot results on unhealthy food, fruit, and vegetable consumption. Overall, in the unhealthy food consumption category, the AI group shows a substantially more improvement compared to non-AI group - (25% improvement, 13.2% deterioration, 0.236 mean) versus (17.5% improvement, 13.8% deterioration, 0.073 mean). In the fruit/vegetable category, we observe somewhat negative changes in both non-AI and AI groups. That is, a higher percentage of participants had negative score changes compared to positive score changes. However, the mean score changes in both groups are very close to zero.

Importantly, we observe participants' behavior changes vary across different types of diabetes-related activities. As a result, a single behavior model may perform poorly in predicting responses to various types of questions. Therefore, we aim to build predictive models that can differentiate behavior changes of three different types: food consumption, physical activities, and diabetes-related knowledge. More specifically, we propose the three models: (i) type-trifecta logistic regression - this model consists of three separate logistic regression components, each produces predictions for responses in one of the aforementioned types; (ii) type-trifecta simple neural network - this model consists of three separate neural net components, each is a simple 2-layer neural net; and (iii) type-trifecta LSTM - this model has a shared LSTM block followed by three separate blocks of linear layers.

Accuracy Evaluation. All of our behavior prediction models were trained on dual Intel E5-2690v4 processors. All experiments were trained in PyTorch using cross entropy loss and the Adam optimizer. For our experiments, we collect all results over 30 random seeds (resulting in different model initializations and test/train splits) and report the mean along with the standard deviation. We remark that there are a lot of missing responses in our training data set (response rates of participants is less than 40%). Therefore, in our experiments, we examine two options: one is to simply encode missing responses as "-1" and the another is to replace missing responses by our model predictions. In addition, we try adding noise (i.e., zero mean Gaussian noise) to participants' responses. The purpose is to examine if this noise helps in improving the robustness of our models or not. Our prediction accuracy results for all models are shown in Tables 1-3, corresponding to three settings: (i) no noise is added to participants' responses and missing responses are encoded as "-1" (Table 1); (ii) similar to (i) but Gaussian noise is added (Table 2); and Gaussian noise is added and missing responses are replaced with the model prediction (Table 3).

All three tables show that differentiating behavior changes according to three different categories of food consumption, physical activities, and knowledge significantly improves the prediction accuracy of our models compared to the baselines. For example, in Table 1, the type-trifecta simple NN model obtains an averaged prediction accuracy of 59% and 58% on the training and test sets, respectively. This is significantly higher than the prediction accuracy of the single simple NN model (i.e., 53.3% and 52.5%). We also observe the similar performance enhancement trend for the logistic regression and the LSTM-based models. In addition, interestingly, unlike in knowledge tracing [Piech et al., 2015] where LSTMbased models are shown to be superior in predicting the knowledge of students, our results show that the type-trifecta simple NN model performs the best. This phenomenon perhaps comes from the limited data availability (with missing responses) in our domain that potentially deteriorates the performance of complex models like LSTM. Lastly, we do not observe a substantial changes in prediction accuracy of all models when we introduce Gaussian noise to responses or insert model predictions to replace missing responses.

8 Conclusion: Results and Learned Lessons

In this work, we developed an RL-based personalized messaging system for diabetes intervention tailored to people living in rural areas where access to healthcare, social support, and education are limited. We ran an extensive field study that involves more than 1000 local villagers participating in our messaging program. Our post-study analysis results show the significant benefit of our approach (compared to the existing system) in the participants' diabetes-related knowledge, physical activities, and high-fat food avoidance. Furthermore, our behavior models which leverage characteristics of participants' responses outperform baselines including the single LSTMmodel that is commonly used in knowledge tracing.

We would like to highlight some important lessons learned from this work. First, collaborations among different partners with different areas of expertise including NGOs, health domain experts, and academics are the key to the success of social impact projects. Second, real-world domains exhibit various challenges that we may not be able to anticipate when building our AI models. Continuing to improve our models and solutions with the adaptation to rising challenges is essential to the long-term impact of the project. For example, our current RL model does not directly account for missing responses from participants. Furthermore, answers about behavior are self-reported and could be misleading. We plan to address these limitations in future work. Third, available realworld data in this domain is extremely limited. Thus, simple models may work better than complex deep learning models.

Acknowledgements

This paper was partially supported by the Google AI for Social Good program and by grant W911NF-20-1-0344 from the US Army Research Office.

Contribution Statement

Sarah Kinsey and Jack Wolf contributed equally to the paper and thus are joint first authors.