Communication is an effective mechanism for coordinating the behaviors of multiple agents, broadening their views of the environment, and to support their collaborations. In the field of multi-agent deep reinforcement learning (MADRL), agents can improve the overall learning performance and achieve their objectives by communication. Agents can communicate various types of messages, either to all agents or to specific agent groups, or conditioned on specific constraints. With the growing body of research work in MADRL with communication (Comm-MADRL), there is a lack of a systematic and structural approach to distinguish and classify existing Comm-MADRL approaches. In this paper, we survey recent works in the Comm-MADRL field and consider various aspects of communication that can play a role in designing and developing multi-agent reinforcement learning systems. With these aspects in mind, we propose 9 dimensions along which Comm-MADRL approaches can be analyzed, developed, and compared. By projecting existing works into the multi-dimensional space, we discover interesting trends. We also propose some novel directions for designing future Comm-MADRL systems through exploring possible combinations of the dimensions.

Background: It is unclear why some physical activity (PA) mobile health (mHealth) interventions successfully promote PA whereas others do not. One possible explanation is the variety in PA mHealth interventions—not only do interventions differ in the selection of persuasive strategies but also the design and implementation of persuasive strategies can vary. However, limited studies have examined the different designs and technical implementations of strategies or explored if they indeed influenced the effectiveness of the intervention. Objective: This scoping review sets out to explore the different technical implementations and design characteristics of common and likely most effective persuasive strategies, namely, goal setting, monitoring, reminders, rewards, sharing, and social comparison. Furthermore, this review aims to explore whether previous mHealth studies examined the influence of the different design characteristics and technical operationalizations of common persuasive strategies on the effectiveness of the intervention to persuade the user to engage in PA. Methods: An unsystematic snowball and gray literature search was performed to identify the literature that evaluated the persuasive strategies in experimental trials (eg, randomized controlled trial, pre-post test). Studies were included if they targeted adults, if they were (partly) delivered by a mobile system, if they reported PA outcomes, if they used an experimental trial, and when they specifically compared the effect of different designs or implementations of persuasive strategies. The study methods, implementations, and designs of persuasive strategies, and the study results were systematically extracted from the literature by the reviewers. Results: A total of 29 experimental trials were identified. We found a heterogeneity in how the strategies are being implemented and designed. Moreover, the findings indicated that the implementation and design of the strategy has an influence on the effectiveness of the PA intervention. For instance, the effectiveness of rewarding was shown to vary between types of rewards; rewarding goal achievement seems to be more effective than rewarding each step taken. Furthermore, studies comparing different ways of goal setting suggested that assigning a goal to users might appear to be more effective than letting the user set their own goal, similar to using adaptively tailored goals as opposed to static generic goals. This study further demonstrates that only a few studies have examined the influence of different technical implementations on PA behavior. Conclusions: The different implementations and designs of persuasive strategies in mHealth interventions should be critically considered when developing such interventions and before drawing conclusions on the effectiveness of the strategy as a whole. Future efforts are needed to examine which implementations and designs are most effective to improve the translation of theory-based persuasive strategies into practical delivery forms.

Mobile health (mHealth) intervention systems can employ adaptive strategies to interact with users. Instead of designing such complex strategies manually, reinforcement learning (RL) can be used to adaptively optimize intervention strategies concerning the user’s context. In this paper, we focus on the issue of overwhelming interactions when learning a good adaptive strategy for the user in RL-based mHealth intervention agents. We present a data-driven approach integrating psychological insights and knowledge of historical data. It allows RL agents to optimize the strategy of delivering context-aware notifications from empirical data when counterfactual information (user responses when receiving notifications) is missing. Our approach also considers a constraint on the frequency of notifications, which reduces the interaction burden for users. We evaluated our approach in several simulation scenarios using real large-scale running data. The results indicate that our RL agent can deliver notifications in a manner that realizes a higher behavioral impact than context-blind strategies.

Background: App-based mobile health exercise interventions can motivate individuals to engage in more physical activity (PA). According to the Fogg Behavior Model, it is important that the individual receive prompts at the right time to be successfully persuaded into PA. These are referred to as just-in-time (JIT) interventions. The Playful Active Urban Living (PAUL) app is among the first to include 2 types of JIT prompts: JIT adaptive reminder messages to initiate a run or walk and JIT strength exercise prompts during a walk or run (containing location-based instruction videos). This paper reports on the feasibility of the PAUL app and its JIT prompts. Objective: The main objective of this study was to examine user experience, app engagement, and users’ perceptions and opinions regarding the PAUL app and its JIT prompts and to explore changes in the PA behavior, intrinsic motivation, and the perceived capability of the PA behavior of the participants. Methods: In total, 2 versions of the closed-beta version of the PAUL app were evaluated: a basic version (Basic PAUL) and a JIT adaptive version (Smart PAUL). Both apps send JIT exercise prompts, but the versions differ in that the Smart PAUL app sends JIT adaptive reminder messages to initiate running or walking behavior, whereas the Basic PAUL app sends reminder messages at randomized times. A total of 23 participants were randomized into 1 of the 2 intervention arms. PA behavior (accelerometer-measured), intrinsic motivation, and the perceived capability of PA behavior were measured before and after the intervention. After the intervention, participants were also asked to complete a questionnaire on user experience, and they were invited for an exit interview to assess user perceptions and opinions of the app in depth. Results: No differences in PA behavior were observed (Z=−1.433; P=.08), but intrinsic motivation for running and walking and for performing strength exercises significantly increased (Z=−3.342; P<.001 and Z=−1.821; P=.04, respectively). Furthermore, participants increased their perceived capability to perform strength exercises (Z=2.231; P=.01) but not to walk or run (Z=−1.221; P=.12). The interviews indicated that the participants were enthusiastic about the strength exercise prompts. These were perceived as personal, fun, and relevant to their health. The reminders were perceived as important initiators for PA, but participants from both app groups explained that the reminder messages were often not sent at times they could exercise. Although the participants were enthusiastic about the functionalities of the app, technical issues resulted in a low user experience. Conclusions: The preliminary findings suggest that the PAUL apps are promising and innovative interventions for promoting PA. Users perceived the strength exercise prompts as a valuable addition to exercise apps. However, to be a feasible intervention, the app must be more stable.

Guideline of Comm-MADRL systems. The guideline positions dimensions where communication influences interaction with the environment and training phases.

Just-in-time adaptive intervention (JITAI) has gained attention recently and previous studies have indicated that it is an effective strategy in the field of mobile healthcare intervention. Identifying the right moment for the intervention is a crucial component. In this paper the reinforcement learning (RL) technique has been used in a smartphone exercise application to promote physical activity. This RL model determines the ‘right’ time to deliver a restricted number of notifications adaptively, with respect to users’ temporary context information (i.e., time and calendar). A four-week trial study was conducted to examine the feasibility of our model with real target users. JITAI reminders were sent by the RL model in the fourth week of the intervention, while the participants could only access the app’s other functionalities during the first 3 weeks. Eleven target users registered for this study, and the data from 7 participants using the application for 4 weeks and receiving the intervening reminders were analyzed. Not only were the reaction behaviors of users after receiving the reminders analyzed from the application data, but the user experience with the reminders was also explored in a questionnaire and exit interviews. The results show that 83.3% reminders sent at adaptive moments were able to elicit user reaction within 50 min, and 66.7% of physical activities in the intervention week were performed within 5 h of the delivery of a reminder. Our findings indicated the usability of the RL model, while the timing of the moments to deliver reminders can be further improved based on lessons learned.